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P RE FACE 

The  Delphi  technique  originated  at  The  Rand  Corporation  in  the  late 
1940s  as  a  systematic  method  for  eliciting  expert  opinion  on  a  variety 
of  topics ,  including  technological  forecasting.  Over  the  years,  Rand 
has  conducted  a  number  of  Delphi  experiments.  In  addition,  hundreds 
of  Delphi  studies  have  been  published  under  corporate,  government,  and 
academic  sponsorship,  covering  a  vast  range  of  topics,  in  che  United 
States  and  abroad,  including  Europe,  the  Soviet  Union,  and  Japan. 

This  report  presents  a  critical  analysis  of  the  Delphi  technique. 

The  analysis  is  in  four  parts.  First,  the  scope  of  the  inquiry  is 
defined,  and  issues  pertinent  to  an  evaluation  of  Delphi  are  raised. 
Second,  conventional  Delphi  Is  evaluated  against  established  professional 
standards  for  opinion  questionnaires,  and  ag  nst  associated  scientific 
standards  for  experimentation  with  huMan  sublets.  Third,  Delphi  is 
evaluated  with  respect  to  Its  assunq>t  ions ,  principles,  and  methodology. 
Fourth,  conclusions  of  the  analysis  are  brought  together  and  recom¬ 
mendations  are  made  for  the  future  use  of  Delphi. 

A  critical  evaluation  of  Delphi  is  long  overdue,  and  It  is  appro¬ 
priate  that  such  an  analysis  be  conducted  by  Rand.  The  research  under¬ 
taken  for  this  report  was  supported  by  the  U.S.  Air  Force  under  Project 
RAND.  Although  portions  cf  the  discussion  Involve  technical  points  of 
sampling,  psychometric  experimental  design,  and  statistical  analysis, 
the  analysis  should  be  generally  useful  to  Air  Force  and  other  planners 
who  may  be  contemplating  the  use  of  Delphi. 
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This  report  presents  a  critical  analysis  and  evaluation  of  the 
Delphi  technique.  To  keep  the  analysis  within  manageable  bounds,  the 
prime  focus  is  on  Delphi  method;  the  diverse  application  areas  are 
secondary.  The  prologue  defines  the  scope  and  organization  of  the  in¬ 
quiry  and  sketches  key  methodological  issues  associated  with  the  com¬ 
plete  cycle  of  conventional  or  characteristic  Delphi  studies.  The  dis¬ 
cussion  proceeds  to  an  evaluation  of  Delphi  against  professional  stan¬ 
dards  for  social  experimentation  and  for  opinion  questionnaires  estab¬ 
lished  by  the  American  Psychological  Association  and  other  national 
professional  organizations.  Analysis  of  conventional  Delphi  indicates 
that  it  does  not  satisfactorily  meet  the  numerous  experimental  and 
methodological  standards  cited  for  test  design,  item  analysis,  subject 
sampling,  reliability,  validity,  administration,  interpretation  of 
findings,  and  warranted  social  use. 

The  main  body  of  the  critique  reviews  methodological  principles 
and  key  assumptions  associated  with  Delphi.  This  analysis  reveals: 
considerable  evidence  that  results  based  on  che  opinions  of  laymen  and 
"experts"  are  indistinguishable  in  many  cases;  aggregate  raw  opinion 
presented  as  systematic  prediction;  technical  shortcomings,  such  as 
•■ntested  and  uncontrolled  halo  effects  in  the  application  of  Delphi 
Questionnaires;  unsystematic  and  non-replicable  definition  and  use  of 
"experts;  manipulated  group  suggestion  rather  than  real  consensus; 
ambiguity  in  results  stemming  from  vague  questions;  acceptance  of  snap 
Judgments  on  complex  issues;  and  the  virtual  absence  of  a  vigorous 
critical  methodological  literature  even  though  hundreds  of  Delphi 
studies  have  been  published.  The  accuracy  of  the  technique,  in  gen¬ 
erating  forecasts  and  other  "expert"  estimates,  is  necessarily  suspect 
so  long  as  Delphi  questions  are  not  empirically  linked  to  objective 
and  independently  verifiable  external  '-alidation  criteria.  These  lia¬ 
bilities  are  counter-balanced  primarily  by  a  popular  demand  for  system¬ 
atic  expert  opinion,  and  by  the  convenience,  low  cost,  and  simplicity 
of  the  method.  It  is  argued  that  such  advantages  are  inconsequential 
if  the  Delphi  concept,  method,  and  results  are  inherently  untrustworthy. 
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The  analysis  concludes  that  conventional  Delphi  is  basically  an 
unreliable  and  scientifically  unvalidated  technique  in  principle  and 
^probably  in  practice.  In  the  absence  of  a  comprehensive  survey  of  the 
extensive  applications  literature,  it  is  suggested,  but  not  pruven,  that 
rhe  results  of  most  Delphi  experiments  are  probably  unreliable  and  in¬ 
valid.  Even  variations  of  conventional  Delphi  should  not  be  encouraged 
unless  they  explicitly  attempt  to  meat  the  challenge  of  generally  ac¬ 
cepted  standards  of  rigorous  empirical  experimentation  in  the  social 
sciences.  Except  for  its  possible  value  as  an  informal  exercise  for  heu¬ 
ristic  purposes,  Delphi  should  be  replaced  by  demonstrably  superior, 
scientifically  rigorous  questionnaire  techniques  and  associated  ex¬ 
perimental  procedures  using  human  subjects. 

As  the  preferred  alternative  to  conventional  Delphi,  professionals , 
funding  agencies,  and  users  are  urged  to  work  with  psychometrically 
trained  social  scientists  who  can  apply  rigorous  questionnaire  tech¬ 
niques  and  scientific  human  experimentation  procedures  tailored  to 
their  specific  needs.  The  final  recommendation  is  that  conventional 
Delphi  be  dropped  from  institutional,  corporate,  and  government  use 
until  its  principles,  methods,  and  fundamental  applications  can  be 
established  experimentally  as  scientifically  tenable. 
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1.  PROLOGUE 


This  report  does  not  attempt  to  review  the  history  of  the  Delphi 
method;  its  primary  purpose  is  to  present  a  critique  of  Delphi.  A 
general  description  of  the  essential  features  of  Delphi  is  presented 
In  the  following  section. 

Organizing  a  meaningful  critique  of  Delphi  presented  many  problems. 
After  considering  various  alternatives,  a  four -step  schema  was  adopted, 
corresponding  to  the  organization  of  this  paper.  First,  raise  tha 
various  types  of  definitive  issues  pertinent  to  a  Delphi  critique. 
Second,  evaluate  conventional  Delphi  against  established  professional 
standards  for  opinion  questionnaires  and  scientific  standards  for  em¬ 
pirical  experimentation  with  human  subjects.  Third,  evaluate  Delphi 
in  terms  of  its  unique  assumptions,  principles,  and  methodology. 

Finally,  summarize  basic  conclusions  and  make  recommendations  concern¬ 
ing  the  future  use  of  Delphi. 

Before  proceeding  with  the  critique,  three  caveats  on  the  scope 
and  limitations  of  this  study  should  be  made  explicit.  Delphi  has  been 
used  for  a  vast  array  of  applications  in  business,  science,  education, 
medicine,  and  other  areas,  both  broad  and  specialized.  The  total  lit¬ 
erature  has  been  estimated  to  include  several  hundred  titles;  a  sub¬ 
stantial  number  of  chese  are  proprietary  or  otherwise  inaccessible. 

The  author  has  been  able  Ko  examine  approximately  150  Delphi  studies 
conducted  at  Rand  and  elsewhere  (see  the  semi-annotated  listing  of 
Delphi  and  related  publications  in  the  Appendix).  The  author  makes 
no  claim  to  having  examined  all  Che  literature,  particularly  all  the 
applications  literature. 

The  focus  of  this  study  is  on  Delphi  principles  and  methodology . 

The  literature  that  has  been  reviewed  contains  the  basic  writings  of 
the  originators  and  key  practitioners  of  Delphi,  both  within  and  out¬ 
side  Rand,  with  critical  coverage  of  Delphi  principles,  assumptions, 
and  procedures.  Evaluative  inferences  from  methodology  to  applica¬ 
tion  are  admittedly  based  on  illustrative  examples  rather  than  on 
direct  examination  of  all  relevant  studies.  The  validity  of  such 
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inferences  should  be  Judged  on  the  coherence  of  arguments  put  forth  and 
the  representativeness  of  examples  used. 

Another  constraint  is  the  elusiveness  of  a  fixed,  universally 
agreed  upon,  working  definition  of  Delphi.  Many  variants  have  emerged, 
some  departing  widely  from  the  Delphi  procedure  associated  with  Its 
Rand  origins.  An  attempt  is  made  in  the  next  section  to  present  a 
definition  and  characterization  of  "conventional  Delphi."  The  use  of 
the  term  "Delphi"  in  this  report  refers  primarily  to  "conventional 
Delphi,"  which  may  or  may  not  apply  to  Delphi  variants,  depending  upon 
the  issues  and  the  context. 

A  third  caveat  on  the  scope  of  this  study  is  that  it  does  not  com¬ 
pare  Delphi  systematically  with  competing  techniques,  A  comparison  of 
Delphi  with  such  techniques  as  simulation,  trend  extrapolation,  gaming, 
morphological  models,  scenarios,  relevance  trees,  input-output  tables, 
contextual  mapping,  brainstorming,  dialectical  planning,  critical  path 
methodology,  etc.,  would  require  an  independent  review  and  evaluation 
of  each  of  these  techniques  and  the  systematic  comparison  of  each  with 
the  others  for  key  objectives  and  application  areas.  Undoubtedly,  auch 
a  comprehensive  critical  appraisal  of  the  methodology  of  the  entire 
field  of  forecasting  and  planning  techniques  la  long  overdue,  for  much 
the  same  reasons  that  an  in-depth  Delphi  critique  is  overdue.  (For 
an  instructive  initial  comparison  and  rating  of  these  and  related  tech¬ 
niques,  and  for  an  appreciation  of  the  magnitude  of  the  task,  see 
Posove,  1967,  and  Sackman  and  Citrenbaum,  1972).  However,  such  an 
appraisal  would  involve  an  effort  an  order  of  magnitude  larger  chan 
che  present  project.  As  desirable  as  such  an  undertaking  might  be, 
this  evaluation  is  necessarily  limited  to  a  comparison  of  conventional 
Delphi  with  scientific  questionnaire  development  and  experimental 
methodology  with  human  subjects,  and  to  questioning  many  of  the  basic 
assumptions  and  methods  of  the  technique  as  it  Is  currently  being  ap¬ 
plied. 


2.  DELPHI  METHOD:  ISSUES,  PROBLEMS,  AND  DEFINITION 


The  objective  in  this  section  is  to  identify  certain  methodological 
Issues,  and  to  characterise  "conventional  Delphi"  for  the  purposes  of 
this  critiq  ie.  The  chronological  framework  for  a  Delphi  study  follows 
a  problem- solving  sequence:  establishment  of  objectives,  formulation 
of  the  problem,  solution  testing,  and  the  writeup  and  dissemination 
of  results.  In  the  Delphi  context,  objectives  Include  needs,  goals, 
basic  value  assumptions,  and  expected  payoffs.  Formulation  of  the 
problem  is  accomplished  through  the  design  of  the  questionnaire  and 
its  experimental  implementation.  Solution  testing  includes  Iterative 
field  administration  and  scoring  of  responses  to  the  questionnaire. 

The  last  stage  involves  the  interpretation  of  results  by  the  Delphi 
director  in  communicating  findings  to  others.  Each  stage  is  briefly 
examined  to  provide  a  chronological  chain  of  methodological  Issues  as 
a  framework  for  this  evaluation. 

2.1.  Delphi  Objectives 

Early  Delphi  studies  at  Rand  were  primarily  concerned  with  sci¬ 
entific  and  technological  foracaating.  They  were  viewed  as  experiments 
with  what  was  thought  to  ba  an  interesting,  and  possibly  useful,  near 
technique.  From  these  humble  beginnings,  Delphi  has  spread  rapidly, 
with  hundreds  of  studies  appearing  in  tha  United  States,  accompanied 
by  growing  uae  in  other  countries,  including  extensive  use  in  the 
United  Kingdom  (Currill,  1972),  recent  uae  in  the  Soviet  Union 
(Martino,  1973),  and  in  Japan.  Delphi  applications  have  grown  in  all 
directions  to  Include  forecasting  of  many  Boclal  phenomena,  including 
human  attitudes  and  values  (Relsman  et  al.,  1969),  and  even  the 
"quality  of  life"  (Dalkey,  Rourke,  Lews  and  Snyder,  1972).  A  large 
and  growing  roster  of  major  firms  have  used  Delphi  for  diverse  purposes 
(see  bibliographic  Appendix).  Applications  have  expanded  until  they 
are  virtually  indistinguishable  from  the  questionnaire  technique, 
broadly  conaidered.  Advocates,  such  as  Turoff  (1971),  have  expanded 
the  scope  of  Delphi  as  a  general-purpose  vehicle  for  distributed  human 
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commun i cat  ion  and  consensus;,  and  for  group  problem-solving.  Delphi  has 
been  propelled  at  an  increasingly  accelerated  rate  into  the  general 
field  of  questionnaire  design  and  development  not  only  lor  "experts," 
but  for  non-experts  as  well.  The  core  question  arises,  how  does  Delphi 
rate  in  comparison  with  competing  approaches  in  the  well  established 
fields  of  questionnaire  design  and  application  in  the  social  sciences? 

The  payoff  of  a  Delphi  study  is  typically  a  presentation  of  ob¬ 
served  expert  concurrence  in  a  given  application  area  where  none  ex¬ 
isted  previously.  This  assumes  that  participating  panelists  are  experts 
in  the  subject  area,  and  that  the  reported  consensus  was  obtained 
through  reliable  and  valid  procedures.  Proponents  of  Delphi  (Dalkey, 
19t>9)  stress  three  quintessential  attributes  that  contribute  to  authen¬ 
tic  consensus  and  valid  results--anonymity  of  panelists,  statistical 
response,  and  iterative  polling  with  feedback.  Is  the  trust  placed  in 
these  central  assumptions  warranted? 

In  any  decision  to  use  delphi,  there  are  various  cost-effectiveness 
considerations.  How  much  does  a  Delphi  study  cost  in  time  and  effort 
for  ttie  director  and  panelists,  and  how  are  such  investments  related 
to  tlie  usefulness  of  the  final  results?  An  associated  issue  is  the 
attractiveness  of  Delphi  as  a  quick  and  easy  way  to  solicit  rational 
expert  opinion  in  an  unknown  area.  Do  such  positive  payoffs  exist? 

2.2  r o rmulation  of  the  Problem 

The  next  step  in  a  Delphi  study  is  the  formulation  of  the  problem, 
the  design  of  the  questionnaire  and  its  application.  How  effectively 
is  the  area  of  inquiry  defined  and  delirdted  by  the  Delphi  investigator? 
Is  there  an  effort  to  make  questionnaires  bias-free?  Are  his  assump¬ 
tions  spelled  ouc?  Are  there  explicit  hypotheses  and  are  they  opera¬ 
tionally  defined?  Has  the  relevant  literature  been  reviewed  and  system¬ 
atically  evaluated?  Have  baseline  statistics  and  qualitative  character¬ 
istics  of  the  area  of  inquiry  been  documented  and  spelled  out,  so  that 
respondents  derive  their  forecasts  and  opinions  from  a  common  specifi¬ 
cation  of  the  current  stete-of-the-art ? 

In  developing  the  questionnaire,  many  technical  considerations 
arise.  Is  the  questi -’it*ire  an  informal,  ad  hoc  collection  of  items, 


or  is  li*  systematically  designed  as  a  standardized  Instrument  to  be 
administered  under  rigorously  controlled  conditions?  How  are  the  items 
constructed?  How  large  was  the  original  pool  of  items,  how  were  they 
derived,  and  what  pilot  procedures  were  used  for  item  analysis  to  prune 
them  down  to  the  final  set  used  for  the  study?  What  psychometric  scal¬ 
ing  approach  was  selected  (e.g.,  Thurstone,  Likert,  or  Guttman  psycho¬ 
metric  scales,  or  econometric  scales,  see  Pill,  1971)  and  what  factors 
determine  the  selection? 

Then  there  are  problems  concerning  the  panelist  sample  to  which 
the  questionnaire  is  applied.  What  is  an  "expert"  in  the  target  ap¬ 
plication  field,  and  how  are  such  experts  operationally  defined?  How 
many  panelists  are  used,  and  what  are  the  expected  levels  of  statisti¬ 
cal  precision  of  the  results  relative  to  planned  sample  size  for  the 
dispersion  of  responses  anticipated?  Can  the  selected  panelist  sample 
be  systematically  related  to  an  objectively  defined  population  with 
measurable  sampling  parameters?  Is  the  choice  of  experts  random,  or 
is  it  selective?  Are  sampling  procedures  rigorously  defined  (see 
Cochran,  1963)  relative  to  hypothesis  testing  for  opinion  polling? 

2.3  Solution  Testing 

In  administering  the  questionnaire,  many  problematic  issues  arise. 
How  are  dropouts  handled  in  the  results?  Which  items  should  be  dropped, 
modified,  or  retained  in  their  original  form  in  successive  Delphi  rounds? 
What  kind  of  feedback,  how  much  feedback  and  in  what  form,  should  be 
presented  to  panelists?  When  is  the  point  of  diminishing  returns  reached 
in  successive  iterations?  How  long  should  the  Intervals  b  between  suc¬ 
cessive  rounds,  and  hew  can  participants  be  encouraged  to  respond 
promptly  to  expedite  turnaround  time?  What  is  the  tradeoff  between 
more  items  and  a  longer  form  versus  fewer  items  with  less  data  in  re  la 
tlon  to  study  objectives?  Does  the  director  reinforce  and  encourage 
conformist  or  dissenting  behavior  in  successive  rounds?  In  working 
with  distributed  Delphi  by  mailed  questionnaires  and  iterative  polling, 
what  opportunities  exist  for  misusing  the  technique? 
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2.4.  Interpretation  and  Use  of  Results 

In  the  final  stage  of  writeup  and  dissemination  of  results,  the 
sain  problems  center  around  the  analysis  and  Interpretation  of  findings. 
Should  only  descriptive  results  be  presented,  or  should  all  statistics 
be  accompanied  by  standard  errors  of  estimate,  clearly  indicating  the 
empirical  level  of  precision?  Is  it  misleading  to  present  only  inter¬ 
quartile  ranges  in  graphic  portrayal  of  Delphi  results,  or  should  the 
full  range  and  true  dispersion  of  results  also  be  presented?  Should 
first-round  results  be  presented  showing  the  full  dispersions  of  ex¬ 
pert  opinion?  How  strongly  should  the  expert  halo  effect  be  exploited, 
or  should  It  be  controlled  in  evaluating  results?  Should  the  procedure 
and  the  interpretation  give  weight  to  adversary  or  consensus  positions? 

How  strongly  should  procedural,  administrative,  statistical,  and 
experimental  limitations  be  stressed  in  the  final  publication?  Are 
results  put  forth  as  scientific  prediction  or  as  conglomerate  opinion? 
Has  provision  been  made  for  replication  testing  or  validity  generaliza¬ 
tion  in  follow-on  studies? 

2.5.  Definition  of  Conventional  Delphi 

The  above  review  of  the  Delphi  cycle  provides  a  backdrop  for  the 
characterization  of  "conventional  Delphi"  as  it  is  used  in  this  cri¬ 
tique.  These  are  briefly  described  below  under  the  categories  of  ob¬ 
jectives,  subjects,  and  techniques. 

The  application  objective  of  conventional  Delphi  may  be  the  fore¬ 
casting  of  specified  events,  long-term  or  short-term;  it  may  be  the 
generation  of  quantitative  estimates  (e.g.,  costs,  market  demand,  num¬ 
bers  of  users,  etc.)  from  a  sit  of  participants;  or  it  may  be  aimed  at 
qualitative  evaluations  (e.g.,  qualitative  scales  of  agreement,  dis¬ 
agreement,  prefev«r.cc-s  among  alternatives).  The  range  of  application 
objectives  thus  im  ludw  type  of  quantitative  or  qualitative  rating 
scale,  and  as  such  is  coextensive  with  questionnaires  broadly  con¬ 
sidered. 

Other  key  objectives  for  conventional  Delphi  may  be  singled  out. 
including  consensus  of  participants  and  heuristic  goals.  The  consensus 
Intent  of  Delphi  is  typically  oriented  toward  controlled  and  rational 


exchange  of  Iterated  opinion  leading  toward  optimal  covergence  of  opin¬ 
ion  achievable  within  the  framevork  of  the  technique.  The  heuristic 
objective  views  Delphi  ao  an  educational  technique  to  help  participants, 
the  director,  and  us...'  to  explore  a  problem  area  more  thoroughly, 
leading  to  greater  Insight  on  the  target  problem. 

Turning  now  to  subjects,  conventional  Delphi  is  primarily  concerned 
with  experts,  but  may  also  uoe  other  subject  groups  who  may  be  informed 
to  a  greater  or  lesser  extent  In  the  target  area  of  inquiry,  but  who 
do  not  qualify  as  experts.  Although  this  report  focuses  on  the  Delphi 
concept  of  expert,  it  is  also  directed  at  the  growing  use  of  non¬ 
experts.  More  broadly,  this  critique  is  concerned  with  the  operational 
sampling  procedures  used  in  selecting  Delphi  subjects,  expert  or  other¬ 
wise. 

The  technique  category  Is  the  most  detailed.  Conventional  Delphi, 
as  used  in  this  report,  exhibits  the  following  characteristics: 

a.  The  format  is  typically,  but  not  always,  a  paper-and-pencll 
questionnaire;  it  may  be  administered  by  mail,  in  a  personal 
Interview,  or  at  an  interactive,  online  computer  console.  The 
basic  data  presentation  and  data  collection  technique  is  the 
structured,  formal  questionnaire  in  each  case. 

b.  The  questionnaire  consists  of  a  series  of  items  using  similar 
or  different  scales,  quantitative  or  qualitative,  concerned 
with  study  objectives . 

c.  The  questionnaire  Items  may  be  generated  by  the  dir2ctor, 
participants,  or  both. 

,d.  The  questionnaire  Is  accompanied  by  some  set  of  instructions, 
guidelines,  and  ground  rules. 

e.  The  questionnaire  is  administered  to  the  participants  for  two 
or  more  rounds;  participants  respond  to  scaled  objective  items; 
they  may  or  may  not  respond  to  open-end  verbal  requests. 

f.  Bach  Iteration  is  accompanied  by  some  form  of  statistical 
feedback  which  usually  Involves  a  measure  of  central  tendency, 
same  measure  of  dispersion,  or  perhaps  the  entire  frequency 
distribution  of  responses  for  each  item. 


g.  Each  Iteration  say  or  My  not  be  accompanied  by  selected  verbal 
feedback  from  some  participants ,  with  the  types  and  mounts 

_ of  feedback  determined  by  tha  director. 

h.  Individual  responses  to  items  are  kept  ai.^aymous  for  all  iter¬ 
ations.  Hovever(  the  director  My  list  participants  by  name 
and  affiliation  as  part  of  the  study. 

I.  Outliers  (l.e.,  upper  and  lower  quart lie  responses)  My  be 
asked  by  the  director  to  provide  written  justification  for 
their  responses. 

J.  Iteration  with  the  above  types  of  feedback  is  continued  until 
convergence  of  opinion  or  "consensus"  reaches  sane  point  of 
diminishing  returns,  as  determined  by  the  director. 

k.  Participants  do  not  meet  or  discuss  Issues  face-to-face,  and 
they  may  be  geographically  remote  from  one  another. 

It  should  be  apparent  that  a  one-sentence  or  even  one-paragraph 
definition  of  "conventional  Delphi"  is  not  possible  without  leaving 
out  many  significant  details  and  quallficaMoas  that  recalve  substan¬ 
tial  attention  in  this  report.  Generally  speaking,  the  working  defini¬ 
tion  of  Delphi  for  this  study  embodies  the  "quintessential"  model  orig¬ 
inating  at  Rand,  with  many  related  variations  that  more  or  less  follow 
the  iterative  questionnaire  format  with  anonymous  statistical  feedback. 

This  completes  the  review  of  issues  raised  by  the  conventional 
Delphi  cycle  and  permits  on  evaluative  comparison  of  Delphi  with  pro¬ 
fessional  standards  for  opinion  questionnaires  and  experimentation 
with  human  subjects. 
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3.  DELPHI  VERSUS  SOCIAL  SCIENCE  STANDARDS 


This  section  presents  key  standards  in  professional  questionnaire 
design  and  use,  and  shows  how  Delphi  measures  up  to  them.  The  evalua¬ 
tive  criteria  are  quoted  frot:  "Standards  for  Educational  and  Psycholog¬ 
ical  Tests  and  Manuals,"  published  by  the  American  Psychological  As¬ 
sociation,  (1966).  This  publication  was  jointly  prepared  by  a  committee 
representing  three  national  organizations:  The  American  Psychological 
Association,  the  American  Educational  Research  Association,  and  the 
National  Council  on  Measurement  in  Education.  This  committee  worked 
over  a  period  of  five  years  in  conjunction  with  numerous  measurement 
specialists  and  test  publishers. 

The  manual  is  currently  undergoing  revision  under  the  auspices  of 
the  APA  Office  of  Scientific  Affairs.  Public  hearings  on  the  proposed 
draft  have  been  held  in  Washington,  D.C.  The  provisional  table  of 
contents  of  the  proposed  version  is  shown  In  Table  1. 

It  cannot  be  too  strongly  emphasized  that  these  guidelines  repre¬ 
sent  responsible  efforts  to  establish  exemplary  scientific  standards 
in  a  controversial  area  with  a  history  of  continuing  abuse  on  the  part 
of  some  test  developers,  and  with  a  history  of  continuing  misunder¬ 
standing  and  under-education  on  the  part  of  the  public.  Whether  or  not 
the  reader  identifies  himself  as  a  social  scientist,  he  should  be  aware 
that  there  is  a  vast  and  highly  germane  literature  reflecting  an  or¬ 
ganized  professional  effort  to  serve  the  public  interest. 

Buros  (1965),  after  dedicating  a  distinguished  lifetime  to  pro¬ 
fessional  quality  control  in  the  public  domain  for  the  testing  field, 
concluded  that  only  partial  success  is  possible  with  the  inevitable 
collusion  between  test  promoters  and  a  gullible  public  that  expects 
far  more  from  tests  than  they  can  possibly  deliver.  The  carryover  to 
Delphi,  as  this  report  shows,  is  more  than  mere  coincidence.  In  the 
absence  of  any  tradition  with  such  guidelines,  Delphi  practitioners, 
participants,  and  users  can  neglect  such  standards  only  at  their  own 
peri  1. 

Some  may  still  argue  that  Delphi  is  not  a  conventional  test. 
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Table  1 

STANDARDS  FOR  DEVELOPMENT  AND  USE  OF 
EDUCATIONAL  AND  PSYCHOLOGICAL  TESTS 


TABLE  OF  CONTENTS 


INTRODUCTION 

Tests  and  Test  Uset>  to  Which  Standards  Apply 

Information  Standards  as  a  Guide  to  Test  Developers 

Procedural  Standards  as  a  Guide  to  Test  Users 

Three  Levels  of  Standards 

The  Audience  for  These  Standards 

Cautions  to  be  Exercised  in  Use  of  Standards 

STANDARDS  FOR  TESTS,  MANUALS,  AND  REPORTS 

A.  Dissemination  of  Information 

B.  Aids  to  Interpretation 

C.  Administration  and  Scoring 

D.  Norms  and  Scales 

STANDARDS  FOR  REPORTS  OF  RESEARCH  ON  RELIABILITY  AND  VALIDITY 

F..  Validity 

Criterion-Related  Validities 
Content  Validity 
Construct  Validity 

Interdependence  of  Validity  Information 

F.  Criterion-Related  Validity 

G.  Reliability 

General  Principles 
Comparability  of  Forms 
Internal  Consistency 
Comparisons  Over  Time 

STANDARDS  FOR  THE  USE  IF  TESTS 

H.  User  Qualification 

I.  Choice  of  Teat  or  Method 

J.  Administration  and  Scoring 

K.  Interpretation  of  Scores 

L.  Standards  for  Test  Use  in  Program  Evaluation 
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Chough  1C  usually  assumes  Che  form  of  an  iterative  paper-and-pencil 
questionnaire.  As  such,  they  argue,  Delphi  is  exempt  and  the  APA  Guide¬ 
lines  should  not  apply.  A  cursory  review  of  the  selected  items,  how¬ 
ever,  reveals  that  Che  guidelines  deal  with  bedrock  questions  concerning 
sampling,  controls,  reliability  of  measures,  and  criterion  validity 
which  are  universal  to  all  scientific  experimentation  with  human  sub¬ 
jects.  If  Delphi  is  to  be  treated  seriously  as  a  professional  tech¬ 
nique,  it  must  be  judged  by  basic,  minimum  standards  applicable  to  all 
empirical  social  science. 

The  historical  precursors  of  Delphi  in  the  opinion  polling  and 
social  psychological  literature  were  most  explicit  in  applying  rigorous 
questionnaire  design  and  sampling  techniques  against  the  methods  and 
findings  of  their  studies.  Cantril  (1938)  and  McGregor  (1938),  in  in¬ 
dependent  studies  on  predictions  of  social  events,  emphasized  the 
severe  limitations  of  questionnaire  format  and  procedures  and  in  the 
representativeness  of  Subject  sampling  for  any  generalizations  of  their 
results.  Kaplan,  Skogstad,  and  Girshick,  In  a  landmark  study  on  "The 
Prediction  of  Social  and  Technological  Events"  (1950),  presented  a 
detailed  listing  of  sampling,  reliability,  and  validity  problems  en¬ 
countered  in  this  field  in  relation  to  rigorous  questionnaire  and  poll¬ 
ing  standards.  As  a  direct  historical  offshoot  of  these  pioneering 
efforts,  the  Delphi  technique  does  not  possess  or  warrant  any  special 
dispensation  exempting  it  from  such  scientific  standards. 

Delphi  proponents  may  protect  that  concern  with  experimental 
method  in  the  application  of  Delphi  questionnaires  is  "misguided"  be¬ 
cause  Delphi  is  a  tool,  and  a  tool,  once  developed,  does  not  have  to 
be  experimentally  administered  each  time  it  is  used.  This  may  be  fine 
for  weighing  scales,  rulers,  compasses,  spectrometers,  voltmeters,  and 
other  measurement  instruments  frequently  used  in  the  physical  sciences. 
However,  it  does  not  apply  to  questionnaires,  or  to  paper-and-pencil 
testing  broadly  considered,  or  to  Delphi  in  particular.  A  question¬ 
naire  is  reliable  and  valid  only  to  the  extent  that  it  is  administered 
under  conditions  that  replicate  the  basic  experimental  controls  under 
which  it  was  originally  designed,  tested,  and  validated.  This  means 
that  each  administration  of  the  questionnaire  is  viewed  as  an 


-12- 


experimental  replication  of  operationally  designed  conditions  for  in¬ 
dividual  response,  collection  of  data,  scoring,  and  interpretation. 

The  layman's  failure  to  realize  that  questionnaire  tests  are  replicated 
experiments  leads  to  abuses  of  the  technique,  non-comparability  of  re¬ 
sults,  and  a  general  increase  in  measurement  error  variance. 

Delphi  iteration  of  questionnaires  with  feedback  is  a  definitive 
empirical  experimental  procedure  with  human  subjects  in  its  own  right. 
Neglect  of  standard  experimental  guidelines  may  lead  to  uncontrolled 
variations  in  results  and  inability  to  define,  replicate,  and  validate 
methods  and  findings.  This  rsglect  may  be  acceptable  for  an  informal 
exploratory  technique,  but  it  is  unacceptable  for  a  rigorous  social 
science  experiment.  The  compounding  of  methodological  problems  gen¬ 
erated  by  an  unscientific  approach  to  the  conduct  of  Delphi  studies  is 
described  and  illustrated  in  this  section. 

3.1.  Scope  of  Standards 

While  the  standards  are  quoted  verbatim  from  the  current  manual, 
the  author  is  fully  responsible  for  the  evaluative  Delphi  commentary. 
The  manual  covers  paper-and-penci 1  testing  broadly  considered  and, 
obviously,  many  of  the  standards  do  not  pertain  directly  to  Delphi. 

In  what  follows,  a  representative  subset  of  key  standards  relevant  to 
Delphi  is  cited,  accompanied  by  evaluative  commentary.  The  citations 
cover  introductory,  interpretive,  validity,  reliability,  and  adminis- 
trative/scorl-ng  standards,  taken  from  applicable  sections  in  the  APA 
manual . 

tn  the  direct  quotes  that  follow,  material  is  reproduced  verbatim 
except  for  one  term.  The  word  "manual"  is  replaced  by  "test  documenta¬ 
tion."  This  is  done  because  it  was  found  that  individuals  unfamiliar 
with  psychometrics  found  it  difficult  to  understand  the  scope  and  In¬ 
tent  of  test  "manuals."  "Test  documentation,"  which  refers  to  test 
materials,  instructions,  controls,  and  reports  of  empirical  results, 
norms,  interpretations,  and  recommendations  for  use,  is  less  likely  to 
cause  unintentional  confusion  for  the  layman  in  relating  the  guidelines 
to  Delphi. 

In  the  lntoduction,  the  manual  states: 
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"These  recommended  standards  cover  not  only  tests  as 
narrowly  defined,  but  also  most  published  devices  for  diag¬ 
nosis,  prognosis,  and  evaluation  ..." 

"The  present  standards  apply  to  devices  which  are 
distributed  for  use  as  a  basis  for  practical  Judgments 
rather  than  solely  for  research.  Most  tests  which  are  made 
available  for  use  in  schools,  clinics,  and  industry  are  of 
this  practical  nature"  (p.  3). 


Conventional  Delphi  studies,  as  applied  prognostications,  or  as  predic¬ 
tions,  or  as  predictions  of  technological  and  social  developments  for 
a  variety  of  end  users,  fall  under  the  general  purview  of  the  manual. 


3.2.  Interpretive  Standards 

Frcm  the  section  on  Interpretation  of  findings,  two  items  are 
selected.  Ratings  accompany  each  standard  listed  in  the  manual.  Rat¬ 
ings  are  "essential,"  "very  desirable,"  or  "desirable,"  shown  in  caps 
at  the  end  of  each  item. 

B4.2.  When  the  statistical  significance  of  a  relationship 
is  reported,  the  statistical  report  should  be  in  a  form 
that  makes  clear  the  sensitivity  or  power  of  the  signif¬ 
icance  teat.  ESSENTIAL  (p,  11). 


Statistical  significance  is  rarely  reported  in  Delphi  studies, 
either  for  precision  of  estimates  or  for  tests  of  the  significance  of 
mean  or  median  differences  between  two  or  more  forecasts.  Consensus 
and  precision  are  implied  from  suggestive  graphs,  not  from  standard 
errors  of  estimates.  With  small  samples  and  large  dispersions,  many 
forecasts  do  not  differ  significantly  from  one  another,  but  are  shown 
to  do  so  by  implication  if  not  by  explicit  statement. 

BA. 4  The  test  documentation  should  state  clearly  what  in¬ 
terpretations  are  intended  for  each  subscore  as  well  as  for 
the  total  test.  ESSENTIAL  (p.  12). 


This  standard  is  especially  pertinent  to  Delphi  studies  where  fore-  ! 

casts  are  made  on  a  broad  and  diverse  target  area.  Each  forecast  should 
be  individually  and  separately  tested  for  dispersion  of  consensus, 

| 

1 

| 
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systeraatic  correlations  with  other  items ,  and  for  significance  of  fore¬ 
casted  differences  against  other  items,  as  is  done  with  quantitative 
scores  in  conventional  questionnaire  item  analyses  (Anastasi,  1968). 

The  author  has  never  seen  the  full  three-dimensional  matrix  of 
items  versus  panelists  versus  rounds  analyzed  by  a  common  statistical 
vehicle,  such  as  analysis  of  variance,  to  test  for  main  and  interaction 
effects.  Nor  are  items  compared  for  homogeneity  of  variance,  linearity, 
and  type  of  empirical  frequency  distributions  for  applying  such  tests. 
With  9raall  samples,  interquartile  Delphi  graphs  are  no  substitute  for 
rigorous  statistical  testing  of  individual  Items  and  item  subsets. 

This  is  not  a  pedantic  frill — differential  statistical  reliability 
requires  differential  interpretation  of  findings. 

Except  for  a  study  by  Derian  and  Morize  (1973),  the  author  has  not 
seen  a  factor  analysis  of  Delphi  Items,  also  part  of  the  standard  re¬ 
pertoire  in  test  item  analysis.  Factor  analysis  is  valuable  for  pruning 
out  redundant  items  that  are  highly  lntercorrelated ,  or  "saying  the 
same  thing"  by  eliciting  the  same  response  from  subjects.  This  type 
of  item  "padding"  is  thus  hidden  from  the  end  user  who  interprets  re¬ 
sults  at  face  value. 

If  these  interpretive  standards  were  respected,  quantitative  Delphi 
findings  would  not  be  presented  In  simplistic,  descriptive  form  to 
potential  users.  They  would  then  not  be  taken  at  face  value  by  users 
who  are  unaware  of  statistical  and  sampling  limitations. 

3.3.  Empirical  Validity 

The  next  items  are  drawn  from  the  "Validity"  section  of  the  APA 
manual.  The  keynote  standard  for  this  section  follows: 

Cl.  Test  documentation  should  report  the  validity  cf  the 
test  for  each  type  of  inference  for  which  it  la  recowmendea . 

If  its  validity  for  some  suggested  interpretation  has  not 
been  investigated,  that  fact  should  be  made  clear.  ES¬ 
SENTIAL  (p.  15). 

This  standard  provides  obvious  protection  for  potential  users  of 
test  results  by  requiring  the  test  publisher  to  Indicate  whether  hie 
test  rests  on  his  (vested)  opinion  (face  validity),  indirect  validity 
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(e.g.,  correlations  with  related  areas),  or  store  direct  forms  of  va¬ 
lidity  testing  (e.g.,  empirical  experimentation  or  real-world  perfor¬ 
mance  measurement).  With  Delphi,  panel  opinion  is  reported,  with 
little  or  no  subsequent  effort  to  test  results  against  actual  or  re¬ 
lated  events  (except  for  a  small  number  of  studies  discussed  l.'ter  In 
this  report).  The  results  are  usually  simply  aggregations  of  itera¬ 
tive  opinions.  For  example,  Cordon  and  Helmer  (1964)  went  no  further 
than  to  show  medians  and  quartlles  and  some  descriptive  Bcatterplots 
for  their  classic  forecasting  study,  and  Nanus,  Wooten  and  Borko  (1973) 
simply  show  frequency  distributions  and  list  some  percentages  for 
quantitative  results  in  their  study  of  the  social  impact  of  multina¬ 
tional  computers.  Measures  of  central  tendency  are  put  forth,  however, 
as  systematic  and  concurred  forecasts  of  specified  events  by  experts. 

The  Delphi  method  typically  measures  very  small  sample  attitudes 
toward  future  events  at  a  given  time.  It  does  not  measure  the  events 
themselves,  nor  does  it  Incorporate  systematic  hypotheses  and  empirical 
feedback  from  such  events.  The  leap  from  raw  opinion  to  future  events 
under  these  conditions  is  strictly  an  act  of  faith. 

"Tie  next  selected  standard  is  found  under  "Content  Validity."  It 
refers  to  item  definition  and  item  sampling. 

C3.  If  a  test  performance  is  to  be  Interpreted  as  a  sample 
of  performance  or  a  definition  of  performance  in  some  uni¬ 
verse  of  situations,  the  test  documentation  should  indicate 
clearly  what  universe  is  represented  and  how  adequace  is 
the  sampling.  ESSENTIAL  (p.  15). 

When  an  area  of  Inquiry  has  been  selected  for  a  Delphi  study,  as 
a  first  step  in  determining  content  validity,  has  the  area  been  ade¬ 
quately  formulated  and  defined?  We  rarely  find  systematic  reviews  of 
application  literature  in  Delphi  studies  leading  to  a  careful,  state- 
of-the-art  definition  of  the  target  domain.  Such  reviews  should  ex¬ 
tract  the  beat  of  precursor  studies  and  define  basic  assumptions  and 
bounds  of  the  inquiry.  We  often  encounter  an  amorphous  soc 1 o-techno- 
logical  area  (e.g.,  scientific  advances,  quality  of  life,  etc.)  where 
the  universe  of  situations  may  be  virtually  indistinguishable  from 
future  society  broadly  considered. 
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The  second  step  in  determining  content  validity  is  demonstrating 
that  the  selected  Items  comprising  tae  questionnaire  represent  a  system¬ 
atic  sampling  of  key  elements  of  the  target  area  of  inquiry.  If  a 
particular  problem  area  has  been  chosen  for  a  Delphi  forecast,  has  a 
taxonomy  been  developed  for  subproblems,  embedding  situations,  re¬ 
sources,  and  classes  of  problem-solvers?  If  so,  has  it  been  used  as 
the  basis  for  a  representative  and  comprehensive  selection  of  items? 

For  example,  in  using  Delphi  to  forecast  computer  developments, 
as  was  done  in  Parsons  and  Williams’s  widely  cited  study  (1968),  content 
validity  preparation  would  call  for  a  systematic  taxonomy  of  hardware, 
softwaie,  peripheral  equipment,  communications  and  applications,  per¬ 
haps  along  the  lines  of  the  classification  scheme  used  by  the  Computing 
Reviews  of  the  Association  for  Computing  Machinery.  If  the  entire  com¬ 
puter  field  is  to  be  covered,  or  some  specified  subset,  the  correspon¬ 
dence  between  final  selected  items  and  the  specified  area  should  be 
spelled  out.  Such  taxonomies,  and  such  accountability  in  matching 
items  against  the  target  universe,  are  rarely  seen  in  the  Delphi  lit¬ 
erature  . 

3.4.  Standards  for  Use  of  Experts 

The  next  two  standards  are  the  only  references  in  the  APA  manual 
to  tlie  use  of  experts  in  test  design  and  analysis.  It  should  come  as 
no  surprise  that  the  social  sciences  have  abandoned  the  use  of  experts 
as  an  integral  part  of  scientific  methodology.  In  test  construction 
and  analysis,  the  role  of  experts  in  generating  and  contributing  ques¬ 
tionnaire  items  to  the  initial  item  pool  is  well  recognized,  and  is 
consistent  with  current  practice.  However,  the  use  of  experts  as  the 
principal  and  exclusive  method  for  validating  tests  has  been  discred¬ 
ited.  For  example,  in  World  War  II,  the  unreliable  "expert"  opinions 
of  experienced,  professional  interviewers  were  dropped  in  favor  of  more 
effective  standardized  objective  testing  procedures  (e.g.,  see 
Thorndike's  account  (1949)  of  the  Aviation  Psychology  Program  of  the 
Army  Air  Forces  in  World  War  II). 

Another  example  of  the  uae  of  experts  in  the  field  of  economics  is 
revealing.  Zarnowitz  (1965)  studied  eight  independent  forecasts  cf  the 
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Gross  National  Product  from  1953  to  1963  derived  from  "expert"  opinion. 
The  average  observed  absolute  error  for  experts  was  $10  billion,  or 
about  2  percent  of  the  GNP  during  this  period.  Zamowltz  found  that 
simple  arithmetic  extrapolation  of  the  increase  occurring  in  the  pre¬ 
vious  year  yielded  an  average  absolute  error  of  $12  billion,  effectively 
the  same  as  the  average  expert  prediction.  Zamowltz  conducted  studies 
of  other  economic  indices  and  obtained  similar  results. 

When  we  leave  the  area  of  short-term  forecasting  in  economics, 
where  extensive  baseline  statistical  indicators  are  available,  and  enter 
the  more  nebulous  areas  of  psychological  and  psychiatric  diagnosis  and 
prognosis,  the  record  of  expert  clinical  opinion  is  and  has  been  in  a 
state  of  disarray.  In  "The  Discontent  Explosion  in  Mental  Health," 
Hersch  (1969)  explicated  the  bankruptcy  in  theory  and  practice  of  the 
unscientific  use  of  clinical  experts  in  empirical  research  on  psycho¬ 
therapy  . 

After  reviewing  some  AO  large-scale  programs  involving  man-machine 
system  experimentation  in  his  comprehensive  book  covering  the  work  in 
this  area  since  World  War  II,  Parsons  (1972)  concluded  that  the  reliance 
of  system  designers  on  the  opinions  and  preferences  of  "so-called  ex¬ 
pert  system  operators"  is  "foolhardy."  He  pointed  out  that  such  experts 
.  .  may  provide  suggestive  leads,  but  are  not  reliable  guides,  as 
demonstrated  by  their  repeated  disagreement  with  objective  data" 

(p.  553).  These  examples  are  illustrative  of  the  repeated  failures  and 
frustrations  encountered  in  the  use  of  experts  in  diverse  social 
science  areas. 

C3.1.  When  experts  have  been  asked  to  Judge  whether  items 
are  an  appropriate  sample  of  a  universe  or  are  correctly 
scored,  the  test  documentation  should  describe  the  rele¬ 
vant  professional  experience  and  qualifications  of  the  ex¬ 
perts  and  the  directions  under  which  they  made  their  judg¬ 
ments.  VERY  DESIRABLE  (p.  15). 

Delphi  exercises  guarantee  anonymity  of  individual  responses  to 
encourage  free  expression  of  opinion.  Some  studies  list  the  names  of 
panelists  and,  in  fewer  cases,  list  their  professional  af 1 i 1  t  it  ions . 

The  author  was  not  able  to  find  any  studies  listing  professional 


training  and  scaled  experience  levels  qualifying  each  individual  a9 
possessing  the  skills  required  to  meet  an  objective  criterion  as  an 
"expert."  This  "very  desirable"  Btandard  ia  effectively  neglected  in 
Delphi  practice. 

C3.ll.  When  the  items  are  selected  by  experts,  the  extent 

of  agreement  among  independent  judgments  should  be  reported. 

DKS  I  RAI1LE .  (p.  16). 

This  standard  makes  an  explicit  distinction  between  independent 
and  dependent  expert  Judgment,  which  gets  to  the  heart  of  Delphi  itera¬ 
tion  "with  feedback."  The  first  round  is  basically  designed  to  secure 
Independent  expert  juugment.  The  second  and  successive  rounds  produce 
strictly  correlated,  or  biased,  judgments.  The  use  of  standardized 
statistical  techniques  for  hypothesis  testing  based  on  random  sampling 
assumptions,  which  may  offer  no  major  problems  for  independent  first- 
round  judgments,  becomes  difficult  and  problematic  in  successive 
rounds--a  methodological  shortcoming  that  has  apparently  not  been 
noticed  by  Delphi  practitioners.  All  rationalisations  about  recon¬ 
sidering,  incorporating  new  information,  and  converging  toward  con¬ 
sensus,  cannot  hide  the  fact  that  independent  Judgment  is  destroyed 
once  the  participant  knows  how  others  have  responded  to  each  item.  If 
Delphi  can  make  no  claims  concerning  independent  expert  opinion,  does 
Delphi  feedback  develop  insight  into  the  issues  for  improved  collective 
judgment  in  successive  rounds? 

3.5.  Theoretical  Standards 

The  next  standard  refers  to  long-term  predictions  and  overlaps 
substantively  with  the  notion  of  forecasting. 

C4.41.  If  a  test  is  recommended  for  long-term  predictions, 
but  comparisons  with  concurrent  criteria  only  are  presented, 
the  test  documentation  should  emphasize  that  the  validity 
of  predictions  is  undetermined.  ESSENTIAL  (pp.  17-18). 

Delphi  practice  neglects  long-term  longitudinal  validation,  and 
typically  dissociates  itself  from  any  systematic  comparisons  with  even 
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second-atring  concurrent  criteria  (e.g.,  short-term  interpretations  of 
long-term  trends).  Panelists  often  disagree  over  what  exists  "today" 
and,  with  rare  exceptions,  Delphi  practitioners  nike  no  effort  to  pre¬ 
sent  panelists  with  a  precise  report  on  "where  we  are"  to  establish  a 
baseline  for  projections  into  the  future.  On  both  counts,  for  this 
"essential"  standard,  Delphi  forecasting  results  should  be  explicitly 
presented  to  potential  users  as  conjectures  of  undetermined  validity. 

Delphi  practitioners  object  to  this  conclusion,  pointing  out  that 
Delphi  has  been  proven  "valid"  and  "accurate"  in  a  few  relatively  re¬ 
cent  studies  Involving  almanac- type  items  (Daikey,  1969)  and  for  rela¬ 
tively  short-term  predictions  (Hartino,  1972).  Established  almanac 
items  (e.g.,  population  of  a  city,  or  gross  national  income  at  a  parti¬ 
cular  point  in  time)  are  not  in  any  substantive  way  generalizable  to 
long-range  forecasts.  What  they  share  in  common  is  the  trivial  prop¬ 
erty  that  we  all  can  exercise  opinions  on  each  item,  hardly  a  sound 
basis  for  generalizing  from  simple  descriptive  facts  anchored  In  the 
past  to  complex  events  in  the  future. 

Martino  (1972)  reports  forthcoming  work  coopa. ing  earlier  Delphi 
predictions  with  outcomes.  The  original  estimates  (as  in  the  Gordon- 
Helmer  study,  1964)  were  derived  from  pooled  respondent  opinion,  and 
the  outcomes  were  also  determined  by  pooled  opinion.  The  abuses  of 
such  a  poet  hoc  subjective  approach  should  be  obvious,  leaving  th_ 
central  issue  of  Delphi  validity  and  accuracy  unresolved. 

The  next  standard  applies  to  identification  of  the  characteristics 
of  participating  panelists. 

C5.2.  The  validity  somplfe  should  be  described  in  test  docu¬ 
mentation  in  terms  of  those  variables  known  to  be  related  to 
the  quality  tested,  uch  as  age,  sex,  socio-economic  status, 
and  level  of  educati  a.  Any  selective  factor  determlnng  the 
composition  of  the  sample  should  be  indicated.  ESSENTIAL 
(p.  19). 

Delphi  studies,  having  promised  anonymity  to  participants,  typi¬ 
cally  do  not  report  key  population  characteristics  of  panelists  such  an 
those  cited  ii:  this  standard.  Such  specification  of  "expert"  samples 
would  permit  more  effective  evaluation  of  the  adequacy  of  the  expert 
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sample.  For  example,  a  long-range  forecasting  study  might  benefit 
from  inputs  from  r  atively  youthful  panelists  who  are  more  likely  to 
be  living  In  and  directly  shaping  the  world  they  are  forecasting; 
lower-class  or  minority  members,  if  the  socio-economic  items  cut  across 
their  future;  more  women  panelists,  if  they  are  underrepresented  (e.g., 
Oalkey,  1969;  Borko,  1970;  and  Bedford,  1972;  have  shown  systematic 
quantitative  and  qualitative  differences  by  sex  in  Delphi  responses); 
wider  geographical  distribution  of  panelists,  if  they  are  concentrated 
in  one  or  two  locales;  etc.  The  author  has  not  encountered  any  studies 
where  panelists  have  been  asked  to  provide  detailed  personal  data  for 
sampling  profiles.  Anonymity  can  still  be  honored  if  panelist  charac¬ 
teristics  are  presented  as  statistical  aggregates. 

The  next  standard  applies  particularly  to  the  pitfalls  inherent 
in  the  voluntary  participation  of  Delphi  panelists. 

C5.3.  If  the  validity  sample  is  made  up  of  records  accumu¬ 
lated  haphazardly  or  voluntarily  submitted  by  test  users, 
this  fact  should  be  stated  in  the  test  documentation,  and 
the  test  user  should  be  warned  that  the  group  is  not  a 
systematic  or  random  sample  of  any  specifiable  population. 

Probable  selective  factors  and  their  presumed  Influence 
on  test  variables  should  be  stated.  ESSENTIAL  (p.  19). 

Panelist  dropout  is  one  of  the  well-known  hazards  c>f  Delphi  experi¬ 
mentation.  Delphi  dropout  rates  are  probably  quite  high.  Although 
he  cited  no  empirical  data,  Martino  (1972)  asserted  that  response  rates 
to  first- round  questionnaires  "typically  ran  50  percent  or  less."  In 
the  only  study  the  author  has  been  able  to  find  on  Delphi  dropouts, 
Bedford  (1972)  noted  that  dropouts  in  a  study  on  home  communication 
services  were  less  motivated  to  participate  In  the  study  (i.e.,  drop¬ 
outs  responded  to  fever  questionnaire  items),  and  more  significantly, 
dropouts  were  considerably  more  critical  of  the  overall  study,  the 
utility  of  questionnaire  items,  and  of  the  relative  stress  placed  on 
various  factors  such  as  "lack  of  concern  for  sociological  and  psycho¬ 
logical  considerations." 

There  is  no  question  but  that  some  selective  factors  operate  to 
determine  the  hard-core  group  that  sticks  with  the  study  through  all 
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lteratlons.  The  reasons  nay  be  positive,  such  as  strong  aotlvatlon 
and  interest  in  Che  target  area,  or  negative,  such  as  a  high  proportion 
of  personal  acquaintances  of  the  director,  or  of  those  in  his  profes¬ 
sional  circle.  Perhaps  those  who  disagree  strongly  with  the  design 
and  content  of  the  questionnaire,  and  those  who  question  initial  re¬ 
sults  (as  in  Bedford's  study),  drop  out  more  often  than  those  who  have 
confidence  in  the  study  and  the  procedure,  or  who  play  along  with  mini¬ 
mum  effort.  To  the  extent  that  any  systematic  panelist  sampling  effects 
are  known,  they  should  be  stated  explicitly  and  taken  into  account  in 
the  evaluation  of  results.  If  the  original  expert  sampling  is  unknown, 
and  if  the  dropout  rate  is  also  unknown,  the  sample  on  which  the  final 
results  are  based  is  doubly  suspect.  This  double  indemnity  is  prob¬ 
ably  the  rule,  not  the  exception,  for  Delphi  studies. 

A  recent  memo  sent  to  me  by  Brownlee  Haydon,  illustrates  the  pos¬ 
sibilities  of  serious  social  abuse  of  conventional  Delphi  in  picking 
a  stacked  panel  of  experts  in  a  controversial  area  with  major  vested 
interes ts. 


"If  you  are  a  regular  reader  of  The  Neu  Yorker ,  you 
may  already  have  seen  the  series  entitled  "Annals  of  In¬ 
dustry — Casualties  of  the  Workplace"  currently  appearing 
in  that  magazine.  The  November  12,  1973  installment  de¬ 
scribes  a  classic  case  of  the  misuse  or  perversion  of  the 
Delphi  process. 

"As  I  read  it,  Arthur  D.  Little,  Inc.  has  undertaken 
for  the  Department  of  Health,  Education  and  Welfare  (Oc¬ 
cupational  Safety  and  Health  Administration)  to  use  the 
Delphi  method  to  arrive  at  a  consensus  on  the  proper 
level  of  exposure  to  asbestos  fibres  (2,  3,  12,  30  fibres 
of  greater  than  3  microns  length  per  cubic  centimeter  of 
air)  to  be  established  as  a  government  safety  standard. 
What  is  almost  unbelievable  is  the  choice  of  ’experts" — 
apparently  members  of  the  asbestos  manufacturing  com¬ 
munity  and  their  "medical  experts"  along  with  a  few  (too 
few)  Independent  medical  researchers  in  the  field  of 
asbestos-induced  cancer!' 


Dr,  Selikoff  was  the  only  member  of  the  "expert  health  panel"  in 
the  Delphi  study  who  ".  .  .  had  not  been  a  paid  consultant  of,  or  whose 
investigations  into  asbestos- related  disease  had  not  been  supported  by, 
some  segment  of  the  asbestos  Industry  .  .  In  this  iV.v  r  article. 
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Dr.  Sellkoff  pointed  out,  "And  what’s  the  point  of  guessing  about  the 
biological  effects  of  asbestos  when  mortality  studies  of  asbestos  workers 
have  already  shown  what  the  effect  has  been?"  (Brodeur,  1973,  p.  172). 

The  next  standard  appears  under  the  section  concerned  with  "con¬ 
struct  validity,"  which  refers  to  the  interpretation  of  theoretical 
constructs  on  which  tests  are  based.  This  standard  raises  the  key 
issue  of  accountability  for  the  interpretation  of  Delphi  results. 

C7.1.  The  test  documentation  should  indicate  the  extent  to 
which  the  proposed  interpretation  has  been  substantiated  and 
should  summarize  investigation  of  the  hypotheses  derived 
from  the  theory.  ESSENTIAL  (p.  23). 

This  requirement  is  largely  ignored  in  Delphi  practice  where  a 
descriptive  approach  characterizes  the  presentation  of  results.  The 
reasons,  theories,  and  hypothetical  constructs  of  expert  panelists 
are  covert,  rather  than  overt.  Panelists  are  asked  for  opinions,  and 
the  occasional  rationale  from  panelists  is  typically  very  brief,  uneven, 
and  often  absent  in  final  reports.  This  haphazard  manner  of  collecting 
and  reporting  data  underscores  the  casual  opinionatlve  essence  of 
Delphi.  There  are  many  levels  of  opinions  ranging  from  snap  Judgments 
to  carefully  organized  and  well-defended  documentation  of  positions 
systematically  linked  to  Interpretive  concepts  of  construct  validity. 
Although  Delphi  practitioners  may  point  out  occasional  exceptions, 
snap  judgments  are  apparently  the  rule  for  most  Delphi  questionnaire 
items,  as  shown  below. 

Bedford  (1972)  appears  to  be  the  only  investigator  who  haa  solic¬ 
ited,  classified,  and  analyzed  all  panelist  comments  in  his  Delphi 
study  on  home  communications  services  (for  a  sample  of  1253  responses). 
His  analysis  of  open-end  verbal  responses  has  led  him  to  defect  from 
"traditional  Delphi  with  its  heavy  emphasis  on  statistical  feedback" 
toward  a  structural  adversary  procedure  "stressing  the  Importance  of 
assumptions,  qualifications,  interpretation  of  general  tiends,  and 
criticism  of  co-panelist's  remarks"  (p.  43). 
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3.6.  Questionnaire  Reliability 

The  next  section  In  the  APA  manual  la  concerned  with  test  reliabil¬ 
ity.  The  first  selected  standard  indteates  minimal  statistical  require¬ 
ments  for  reliability  reporting. 

03.  Reports  of  reliability  studies  should  ordinarily  be  ex¬ 
pressed  in  the  test  documentation  in  terms  of  variances  for 
error  components  (or  their  square  roots)  or  standard  errorB 
of  measurement!  or  product-moment  reliability  coefficients. 
ESSENTIAL  (p.  29). 

Delphi  studies  invariably  tend  to  ignore  such  "essential"  consider¬ 
ations  of  test  and  item  reliability.  For  example,  Sahr  (1970)  presents 
some  50  pages  filled  with  descriptive  quantitative  data  comparing  three 
Delphi  studies  conducted  at  the  Institute  for  the  Future.  At  no  point 
does  he  report  a  single  statistic  indicating  "variances,  standard 
errors  of  measurement,  or  product-moment  reliability  coefficients" 
required  by  this  standard.  Dalkey  (1969)  has  made  an  initial  attempt 
in  this  direction  by  irdicatlng  increasing  reliability  of  medians  with 
increasing  sample  size  of  panelists — a  surprise-free  result.  (The 
standard  error  of  measures  of  central  tendency  generally  vary  inversely 
with  the  square  root  of  sample  size.)  He  does  not  present  standard 
errors  of  medians  for  individual  item  results  as  minimally  required 
by  this  standard.  Dalkey  does  present  split-half  (odd-even)  reliabili¬ 
ties  for  some  results  with  coefficients  usually  varying  between  .4  and 
.6.  This  reported  level  of  reliability  is  marginal  for  useful  question¬ 
naires.  Furthermore,  these  are  for  end-results  with  non-independent 
or  feedback-affected  opinions,  as  discussed  earlier.  Reliability  of 
first-round  results  would  provide  more  meaningful  coefficients  for 
rigorous  statistical  testing.  Dalkey 's  attempt  to  measure  reliability 
is  the  exception  rather  than  the  rule  for  the  descriptive  statistics 
characteristic  of  the  Delphi  literature. 

For  example,  Martino  (1972)  atcempts  to  demonstrate  the  reliability 
of  Delphi  by  listing  several  analogous  Items  in  presianably  independent 
studies  which  resulted  in  "similar"  predictions.  No  cot  t  ion  co¬ 

efficients  or  other  statistical  indices  are  reported,  no  account  is 
presented  of  deleted  items  or  discordant  items ,  and  no  attempt  is  made 
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to  describe  comparability  of  test  conditions  for  final  results.  A 
study  by  McLoughlin  (1969)  is  cited  in  which  two  groups  of  experts 
provided  independent  forecasts  for  55  identical  questionnaire  items. 

The  obtained  standard  deviation  of  the  differences  of  the  medians  be¬ 
tween  the  two  groups  wad  3.54  years  for  events  expected  to  occur  be¬ 
fore  1990.  Martino  concludes  that  this  result  shows  a  "high  degree  of 
consistency."  On  the  contrary,  assuming  a  5  percent  level  of  signifi¬ 
cance,  this  finding  means  that  the  "true"  median  forecast  falls  some¬ 
where  between  '  7  years  of  the  obtained  forecast  ('  two  standard  devia¬ 
tions),  which  is  hardly  the  basis  for  claiming  a  "high  degree  of  con¬ 
sistency."  A  95  percent  confidence  belt  of  14  years  is  not  very  good 

for  forecasts  of  events  expected  to  occur  with  20  years. 

The  next  standard  cited  also  applies  to  test  reliability,  in  par¬ 
ticular  the  stability  of  results. 

D6.  Test  documentation  should  indicate  to  what  extent  test 
scores  are  stable,  that  is,  how  nearly  constant  the  scores 
are  likely  to  be  if  a  test  is  repeated  after  time  has  lapsed. 

Test  documentation  should  also  describe  the  effect  of  any 
such  variation  on  the  usefulness  of  the  test.  The  time 
interval  to  be  considered  depends  on  the  nature  of  the  test 
and  on  what  Interpretation  of  the  teBt  scores  is  recommended. 
ESSENTIAL  (pp.  30-31). 

This  "essential"  standard  says,  as  applied  to  Delphi,  that  the 
questionnaire  should  be  replicated  at  a  later  time  on  an  independent 
sample  of  panelists,  following  original  procedures,  so  that  earlier 
results  can  be  compared  with  later  results  to  determine  test  reliabil¬ 
ity  over  time.  No  such  replications  are  reported  in  the  Delphi  litera¬ 
ture.  This  type  of  reliability  is  especially  important  for  Delphi  be¬ 
cause  the  method  presumably  measures  attitudes  toward  the  future  which 
change  to  a  greater  or  lesser  extent  with  changing  conditions  and  in¬ 
dependent  panels.  The  absence  of  such  studies,  and  the  lack  of  inter¬ 
pretations  of  the  underlying  dynamics  of  attitude  changes  toward  the 
future,  is  a  major  methodological  and  theoretical  shortcoming  of  Delphi. 

Some  Delphi  proponents  object  to  a  study  of  the  underlying  dynamics 
of  attitudes  toward  the  future,  as  distinct  from  and  peripheral  to  the 
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domain  of  Delphi  opinion  technology.  The  argument  is  that  opinions 
are  quite  different  from  attitudes,  particularly  if  they  are  concerned 
with  technical  subjects.  Such  a  position  reflects  the  isolation  of 
Delphi  from  the  mainstream  of  social  science.  The  author  concurs  with 
Anastasl  (1968)  who  says  "Opinion  is  sometimes  differentiated  from 
attitude,  but  the  proposed  distinctions  are  neither  consistent  nor 
logically  defensible.  More  often  the  two  terms  are  used  interchange¬ 
ably  .  .  (p.  480).  In  this  report  the  two  terras  are  used  more  or 

less  synonymously. 

The  validity  of  any  testing  instrument  cannot  be  greater  than  its 
reliability;  that  is,  a  test  cannot  correlate  more  highly  with  any 
external  validation  criterion  than  its  correlation  with  Itself  (reli¬ 
ability)  .  If  Delphi  results  prove  unstable  in  a  given  area  over  the 
short  run,  as  with  attitude  fluctuations  over  time,  its  value  as  a 
prognostic  instrument  is  likely  to  be  worthless  over  the  long  run. 
Longitudinal  reliability  studies  of  this  type  are  essential  for  any 
defensible  use  of  Delphi  or  its  derivatives. 

3. 7.  Experimental  Sampling  Standards 

The  final  section  of  the  APA  manual  covers  sampling  scales  and 
norms.  The  next  standard  overlaps  to  some  extent  with  prior  discussion, 
but  is  worth  emphasizing. 

F6.ll.  Norms  reported  in  test  documentation  should  be 
based  on  a  well-planned  sample  rather  that.  data  col¬ 
lected  primarily  on  the  basis  of  availability.  Ei'SE 
TIAL  (p.  35). 

Selection  of  panelists  for  Delphi  studies  tends  to  reflect  ex¬ 
pediency  rather  than  a  "well-planned  sample,"  particularly  when  inves¬ 
tigators  are  not  accountable  for  Baaple  specification  under  the  ano¬ 
nymity  clausa.  Heavy  Delphi  dropout  rates  can  only  compound  and 
aggravate  this  shortcoming. 

The  next  listed  standard  specifically  warns  against  a  standard 
Delphi  practice  of  developing  norma  (generalizations)  from  small  sam¬ 
ples  of  panelists. 
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F6.31.  If  the  sample  on  which  norms  are  baaed  la  small  or 
or  otherwise  undependable,  the  user  should  be  cautioned  ex¬ 
plicitly  in  the  test  documentation  regarding  the  possible 
magnitude  of  errors  arising  in  interpretation  of  scores. 

ESSENTIAL  (p.  36). 

If  Delphi  investigators  made  it  common  practice  to  report  standard 
errors  of  estimates  for  small  samples,  it  would  be  apparent  to  all  that 
higher  levels  of  precision,  larger  samples,  and  well-defined  samples 
would  be  required.  This  is  particularly  true  where  medians  are  re¬ 
ported  rather  than  means,  since  the  standard  error  of  medians  is  usu¬ 
ally  larger  than  mean  errors.  It  is  also  the  case  for  forecasts  far 
into  the  future,  where  observed  dispersions  are  typically  very  large, 
precision  poor,  and  more  extensive  sampling  necessary.  Martino  (1972), 
for  example,  has  demonstrated  an  Increasing  dispersion  of  forecasts 
in  many  Delphi  studies  as  the  expected  year  of  occurrence  is  farther 
away . 

The  next  standard  describes  a  practice  that  has  been  consistently 
neglected  in  the  Delphi  literature. 

F6.4  Test  documentation  should  report  whether  scores  vary 
for  groups  differing  on  age,  sex,  amount  of  training,  and 
other  equally  Important  variables.  ESSENTIAL  (p.  36). 

The  tacit  Delphi  assumption  is  that  the  pooled  opinion  of  experts 
is  better  than  that  of  any  subgroup  of  experts.  This  may  or  may  not 
be  the  case  for  any  given  area  of  Delphi  inquiry.  However,  the  fact 
remains  that  there  may  be  systematic  effects  related  to  the  kinds  of 
sampling  characteristics  mentioned  in  this  standard.  It  behooves  the 
Delphi  investigator  to  test  for  such  effects  and  to  report  them,  rather 
than  to  assume  uncritically  that  the  whole  is  axlomatically  better 
than  any  of  its  parts.  Dalkey  (1969)  has  demonstrated  sex  differences 
for  almanac  Items;  Borko  (1970)  lists  substantial  sex  and  professional 
differences  for  library  and  information  science  research  items;  and 
Derlan  and  Morlze  (1973)  show  systematic  differences  between  types  of 
medical  specialists  (e.g.,  researchers  versus  clinicians)  in  medical 
forecasting. 
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3,8.  Conclusion  for  Social  Science  Standards 

This  concludes  the  tour  through  portions  of  the  APA  manual  of 
standards  relevant  to  Delphi.  It  should  be  abundantly  clear  that  con¬ 
ventional  Delphi  neglects  virtually  every  major  area  of  professional 
standards  for  questlonalre  design,  administration,  application,  and 
validation.  In  no  sense  is  Delphi  found  to  be  a  serious  contender  In 
scientific  questionnaire  development  and  in  the  experimentally  con¬ 
trolled  and  replicable  application  of  questionnaires. 

But  this  is  not  the  whole  story  by  any  means .  Many  key  areas  re¬ 
main  to  round  out  the  picture.  Only  the  methodology  common  to  any 
questionnaire  instrument  has  been  covered.  The  special  characteristics 
of  Delphi  remain  to  be  reviewed  and  evaluated,  and  this  is  the  task 
of  the  next  section. 
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4.  DELPHI  EVALUATION 


The  pre-Delphi  literature,  mentioned  earlier,  anticipated  many  of 
the  evaluative  problems  encountered  in  the  use  of  opinion  to  forecast 
social  and  technological  events.  McCregor  (1938)  and  Cantril  (1938), 
from  social  psychological  approaches,  found  the  forecasting  process 
via  questionnaire  to  provide  a  medium  for  projecting  personal  values 
and  attitudes  of  the  respondents.  They  made  no  claims  for  the  validity 
of  the  technique  in  forecasting  social  events,  nor  for  the  ability  of 
experts  to  predict  compiex  social  events  any  better  than  non-experts. 
McGregor's  conclusion  summarizes  his  findings.  "The  amount  of  informa¬ 
tion  possessed  by  the  predictor,  and  his  sophistication  or  expertness 
are  shown  to  have  little  significance  in  the  determination  of  predic¬ 
tions  concerning  complex  social  phenomena.  The  quality  of  Information 
as  determined  by  ambiguity  and  importance  is  much  more  decisive" 

(p.  203) .  Cantril  obtained  similar  results  and  concluded  that  "When¬ 
ever  the  prediction  of  a  social  event  is  based  wholly  or  in  part  upon 
an  internal  frame  of  reference,  objectivity  is  rare,  if  not  impossible, 
because  of  ego- involvement"  (p.  388).  Both  studies  illustrate  further 
the  difficulties  encountered  in  the  use  of  opinion,  expert  of  other¬ 
wise,  in  predicting  events. 

Kaplan,  Skogstad,  and  Girshick  (1950)  sixmnarlzed  the  difficulties 
they  encountered,  In  trying  to  generalize  from  their  results  in  social 
and  technological  forecasting  by  questionnaire,  as  fundamentally  a 
problem  of  sampling.  They  concluded  that  "...  the  most  serious 
question  raised  by  a  study  of  prediction  is  whether  the  analysis  is 
made  on  a  statistically  stable  population.  The  difficulties  are  three¬ 
fold:  those  concerning  the  group  of  predictors,  those  concerning  the 
questions  asked,  and  those  concerning  procedure,"  (p.  108).  These 
authors  were  skeptical  of  their  findings  because  of  uncontrolled  and 
unknown  individual  differences  between  subjects,  obvious  differences 
between  questionnaire  items  precluding  extrapolations  to  related  areas, 
and  the  limitations  of  the  procedure — such  as  subjective  factors  in 
experimenters'  judgment,  time  constraints  in  selecting  items,  multiple 


choice  and  probabilistic  format  of  itemB,  and  discrepancies  between  the 
use  of  judgment  of  subjects  under  experimental  conditions  compared  with 
the  use  of  experts  under  more  realistic  conditions.  This  was  a  pivotal 
^study,  one  that  provided  key  leads  for  initial  Delphi  developments. 
Unfortunately,  scientific  admonitions  concerning  statistical  representa¬ 
tiveness  and  experimental  rigor,  as  we  have  seen  in  the  previous  sec¬ 
tion,  were  disregarded  by  the  Delphi  originators. 

The  critical  literature  on  Delphi  is  uneven  and  sparse.  Quinn 
(1971)  has  described  limitations  of  forecasting  in  general  that  apply 
to  Delphi,  including  such  factors  as  surprise  events,  inadequate  or 
biased  data,  and  unpredictable  interactions.  Pill  (1971)  explores 
various  limitations  of  Delphi  and,  in  connection  with  its  reliance  on 
human  intuition,  suggests  that  "...  perhaps  the  Delphi  technique 
should  be  less  allied  with  science  than  with  metaphysics"  (p.  61). 
Nilkovich,  Annoni ,  and  Mahoney  (1972)  emphasize  the  loss  of  valuable 
data  because  Delphi  participants  are  not  allowed  to  interact  directly. 
Weaver  (1969,  1970)  suggests  that  Delphi  pays  inadequate  attention  to 
psychological  values  and  attitudes  toward  the  future.  (See  Fishbein, 
1967,  for  a  comprehensive  introduction  to  the  methodological  literature 
on  attitude  testing.)  Morris  (1971)  has  criticized  Delphi  for  not 
capitalizing  on  the  extensive  mathematical  literature  on  the  theory 
of  subjective  probabilities  (e.g.,  Bayesian  analysis).  In  the  pre¬ 
vious  section,  we  have  shown  that  this  criticism  applies  not  only  to 
advanced  probabilistic  analyses,  but  also  to  elementary  statistical 
treatment  of  raw  Delphi  data. 

Derlan  and  Morize  (1973)  criticize  conventional  Delphi  for  taking 
the  central  tendency  of  pooled  opinion  at  face  value  as  a  best  estimate 
of  expert  opinion.  Through  the  use  of  factor  analysis  of  Delphi  par¬ 
ticipants  In  their  study,  they  found  subgroups  of  experts  clustering 
together  with  consistent  opinions.  They  recommend  analyses  of  sub¬ 
groups  rather  than  composite  consensus. 

"However,  rather  than  the  consensus  Itself  which  only  ex¬ 
presses  the  average  opinion  of  the  group,  the  knowledge  of 
the  structure  of  the  answers  and  motivations  specific  to 
the  different  subgroups  constituting  the  panel  of  experts 
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can  be  extremely  useful  to  the  decision  maker.  Thus,  in 
the  case  of  the  artificial  heart  project,  it  is  not  unim¬ 
portant  to  be  able  to  assess  for  instance  the  differences 
in  points  of  view  between  research  specialists,  clinicians 
and  surgeons." 

As  mentioned  earlier,  Bedford  (1972)  found  so  many  shortcomings 
in  conventional  Delphi,  that  he  developed  an  Independent  technique 
called  SPRITE:  Sequential  Polling  and  Review  of  Interacting  Teams  of 
Experts.  In  a  comparative  Delphi  study  on  future  home  communication 
services,  he  found  no  consistent  statistical  differences  in  forecast¬ 
ing  results  between  housewives  and  experts,  and  he  found  the  qualita¬ 
tive  responses  more  useful  than  the  quantitative  results.  This  led 
Bedford  to  drop  the  traditional  Delphi  emphasis  on  consensus,  and  move 
toward  "controlled  conflict"  between  contrasting  groups,  and  to  drop 
statistical  feedback  in  favor  of  qualitative  arguments.  SPRITE  is  an 
example  of  non-conventional  Delphi. 

Weaver  (1972)  has  probably  contributed  the  most  extensive  critical 
review  of  Delphi  uncovered  in  our  survey  of  the  literature.  He  asserts 
that  the  "vast  majority"  of  Delphi  studies  "tend  to  be  uncritical"  and 
"promotional."  He  believes  that  "Delphi  panels  cater  to  the  power 
structure"  (p.  21).  Delphi  studies  reviewed  "suffer  from  technical 
limitations"  subjects  to  experimenter  bias  in  collating  and  summarizing 
responses,  subjectivity,  lack  of  alternatives,  and  no  checks  on  wording 
or  order  of  items.  Weaver  asserts  that  "There  is  serious  sterility  In 
the  process  of  summarizing  mass  information  into  narrowly  terse  state¬ 
ments.  There  is  a  serious  absence  of  any  effort  to  probe  benaath  the 
surface  for  explanations"  (p.  21). 

In  discussing  needed  changes  in  Delphi,  Weaver  makes  several  re¬ 
commendations.  He  suggests  a  shift  away  from  mere  description  of  events 
to  explaining  events.  He  would  drop  anonymity,  statistical  feedback 
of  dates  and  probabilities,  and  "consensus  forcing  procedures."  He 
questions  the  notion  "that  convergence  improves  the  accuracy  of  a  fore¬ 
cast,"  Weaver  would  add  face-to-face  interaction  and  direct  confronta¬ 
tion  to  ensure  exchange  of  assumptions,  arguments,  and  conclusions, 
and  cites  an  example  of  such  an  exercise  conducted  at  the  International 
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Adult  Education  Seminar,  at  Syracuse  University.  Weaver  believes  that 
the  elimination  of  anonymity  and  statistical  feedback,  and  the  Intro¬ 
duction  of  face-to-face  confrontation,  still  represents  a  recognizable 
variant  of  Delphi.  It  seems  to  this  reviewer  that  with  the  rejection 
of  the  three  "quintessential"  elements  of  conventional  Delphi  (ano¬ 
nymity,  iteration,  and  statistical  feedback)  any  resemblance  between 
Weaver's  recommended  interactive  group  process  and  conventional  Delphi 
is  strictly  coincidental.  Weaver's  recommended  approach  closely  re¬ 
sembles  Heller's  method  of  "group  feedback  analysis"  (1969)  which  was 
developed  independently  of  Delphi. 

In  his  summary,  Weaver  asserts  that 

"At  present  Delphi  forecasts  come  up  short  because  there  is 
little  emphasis  on  the  grounds  or  arguments  which  might 
convince  policy-makers  of  the  forecasts'  reasonableness. 

There  are  insufficient  procedures  to  distinguish  hope 
from  likelihood.  Delphi  at  present  can  render  no  rigorous 
distinction  between  reasonable  judgment  and  mere  guesBingi 
nor  does  it  elearly  distinguish  priority  and  value  state¬ 
ments  from  rational  arguments,  nor  feelings  of  confidence 
and  desirability  from  statements  of  probability." 

Weaver  concludes  by  urging  his  recommended  changes  in  conventional 
Delphi  and  by  stressing  its  value  aB  an  educational  and  heuristic  tool, 
as  distingulsed  from  forecasting. 

The  author  generated  a  list  of  advantages  and  disadvantages  of 
Delphi  in  his  review  of  the  literature,  as  a  preparatory  exercise  to 
develop  a  data  base  for  this  critique.  The  disadvantages  soon  vastly 
outstripped  the  advantages.  Approximately  200  negative  criticisms 
were  compiled.  These  were  arrayed  as  ten  key  questions  which  are 
presented  below. 

The  advantages  of  conventional  Delphi,  at  least  in  this  reviewer's 
estimation,  are  primarily  concerned  with  low  cost,  versatile  applica¬ 
tion  to  virtually  any  area  where  "experts"  can  be  found,  ease  of  ad¬ 
ministration,  minimal  time  and  effort  on  the  part  of  the  director  and 
panelists,  and  the  simplicity,  popularity,  and  directness  of  the  method. 
However,  these  and  related  advantages  are  characterise  '.cal ly  obtained 
by  unwarranted  assumptions  in  method  and  approach  and  by  seriously 
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compromising  the  reliability,  validity,  and  integrity  of  final  reaults. 
Such  advantages  are  inconsequential  if  the  conventional  Delphi  concept, 
method,  and  results  are  inherently  tintrustworthy. 

The  ten  key  questions  for  conventional  Delphi  are  these: 

1.  is  the  Delphi  concept  of  the  expert  and  its  claim  to  represent 
valid  expert  opinion  scientifically  tenable,  or  is  it  over¬ 
stated? 

2.  Are  Delphi  claims  of  the  superiority  of  group  over  individual 
opinion,  and  of  the  superiority  of  remote  and  private  opinion 
over  face-to-face  encounter,  meaningful  and  valid  generaliza¬ 
tions? 

3.  Is  Delphi  consensus  authentic,  or  specious  consensus? 

4.  Are  Delphi  questions,  particularly  forecasting  questions, 
precise  and  meaningful? 

5.  Are  Delphi  responses  precise  and  unambiguous? 

b.  Are  Delphi  results  meaningful  and  unambiguous? 

7.  Is  Delphi  primarily  concerned  with  collections  of  snap  Judg¬ 
ment  opinions  of  polled  individuals  from  unknown  samples,  or 
is  it  concerned  with  coherent  predictions,  analyses,  or  fore¬ 
casts  of  operationally  defined  and  systematically  studied  be¬ 
haviors  or  events? 

8.  Does  Delphi  anonymity  reinforce  scientific  accountability  or 
unaccountability  in  method  and  findings? 

9.  Does  Delphi  systematically  encourage  or  discourage  the  adver¬ 
sary  process  and  exploratory  thinking? 

10.  Does  Delphi  represent  a  critical  tradition,  or  is  it  uncrit¬ 
ically  isolated  from  the  mainstream  of  scientific  question¬ 
naire  development  and  behavioral  experimentation,  and  does 
Delphi  set  a  desirable  or  an  undesirable  precedent  for  inter¬ 
disciplinary  science  in  the  professional  planning  and  policy 
studies  community? 

Each  of  these  questions  is  discussed  in  the  ten  sections  that 
follow.  The  answers  to  these  questions,  based  on  this  analysis,  are 
found  in  the  eleventh  and  concluding  section. 


4.1.  Delphi  Experts 

As  emphasized  earlier,  It  Is  almost  Impossible  to  find  current 
psychometric  or  social  science  literature  on  "experts."  For  example, 
the  author  was  not  able  to  find  any  continuing,  systematic  studies  on 
experts  in  the  recent  Psychological  Abe  tracts  (except  for  highly  spe¬ 
cialized  applications  in  legal  testimony  and  clinical  diagnosis),  In 
Derelson  and  S telnet* s  (1964)  inventory  of  findings  on  human  behavior, 
In  the  United  Nation's  Encyclopedia  of  Social  Science  (Gould  and  Kolb, 
1964),  in  the  international  Encyclopedia  of  Social  Science  (Sills, 
1968),  or  in  many  other  social  science  texts  that  he  has  examined. 

Sole  reliance  on  the  use  of  expert  opinion  for  scientific  validation 
has  long  been  discredited.  There  is  a  very  extensive  literature  on 
psychometric  scales  for  judgments,  attitudes,  and  opinions  for  a  va¬ 
riety  of  tests  (e.g.,  see  Anastasl,  1968),  and  for  specified  subject 
populations,  but  not  for  "experts." 

In  assembling  a  relatively  small  group  of  experts,  typical  of 
Delphi,  the  director  is  tempted  to  select  panelists  he  knows,  or  col¬ 
leagues  recommended  by  his  acquaintances.  Such  selection  is  tempting 
because  it  is  easier  and  faster,  with  fewer  rejections.  Perhaps  the 
fastest  way  to  discourage  a  Delphi  study  is  for  the  director  to  fight 
uphill  against  a  high  dropout  rate  from  panelists.  The  resulting 
sample  of  "experts"  is  likely  to  include  people  with  similar  back¬ 
grounds  and  Interests,  who  think  along  similar  lines.  Such  groups  may 
also  tend  to  comprise  an  elite  with  a  vested  Interest  in  promoting  the 
area  under  Delphi  investigation.  Expert  panels  are  often  selected 
from  accessible  experts,  and  this  accessibility  is  largely  covert. 
Delphi  reports  characteristically  offer  little  or  no  information  about 
panelist  selection,  and  provide  no  safeguards  against  such  abuses. 

Top  names  in  the  field  under  investigation  lend  prestige  to  the 
Delphi  study.  The  inclusion  of  prestigious  Individuals  acts  as  a 
magnet  to  attract  others  less  prestigious.  However,  the  prestige 
personalities  may  be  counterproductive — the  younger  and  more  obscure 
panelists  may  be  more  highly  motivated  to  work  harder  at  the  question¬ 
naire  provide  more  carefully  considered  responses.  There  is  always 
the  choice  between  the  older,  established  professional  versus;  the  young 
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Turk.  Representation  of  the  entire  spectrum  is  probably  better  than 
taking  sides,  at  least  to  help  assure  more  diversified  opinion.  Turoff 
(1971)  and  Martino  (1972),  alarmed  by  uncontrolled  panelist  dropout 
rates,  and  concerned  with  the  need  for  higher  levelc  of  panelist  motiva¬ 
tion  and  more  carefully  reasoned  responses ,  recommend  budgetary  provi¬ 
sion  for  honoraria  for  panelist  time  and  effort. 

The  use  of  experts  leads  to  a  serious  technical  limitation  of  the 
Delphi  questionnaire — the  fallacy  of  the  halo  effect,  in  this  case  the 
expert  halo  effect.  This  is  the  tendency  of  respondents  to  be  unduly 
influenced  by  any  favorable  or  unfavorable  characteristic  of  the  ques¬ 
tionnaire  which  colors  and  contaminates  their  judgment.  For  example, 
a  highly  desired  technological  event  may  systematically  receive  more 
optimistic  forecasts  than  a  neutral  event. 

Delphi  is  enmeshed  in  a  pervasive  expert  halo  effect.  The  direc¬ 
tor,  the  panelists,  and  the  users  of  Delphi  results  tend  to  place  ex¬ 
cessive  credence  on  the  opinions  of  "experts."  Panelists  bask  under 
the  warm  glow  of  a  kind  of  mutual  admiration  society.  The  director 
has  the  prestige  of  pooled  authority  behind  his  study,  and  the  uncri¬ 
tical  user  Is  more  likely  to  feel  snug  and  secure  under  the  protective 
wing  of  an  impressive  phalanx  of  experts. 

The  result  of  the  expert  halo  effect  for  Delphi  Is  to  make  no  one 
accountable.  The  director  merely  reports  expert  opinion  objectively, 
according  to  prescribed  procedure;  he  is  not  responsible  or  liable  for 
outcomes.  The  panelist  obligingly  follows  the  ritual,  protected  at  all 
points  by  faceless  anonymity.  The  user  can  always  claim  that  he  was 
simply  following  the  best  advice  available,  and  that  he  is  not  respons¬ 
ible  for  what  the  experts  say.  Everyone  has  an  out,  no  one  needs  to 
take  any  serious  risks,  and  no  one  is  ultimately  accountable.  With 
so  much  to  gain,  so  little  to  Invest  at  such  low  risk,  no  wonder  the 
method  Is  so  popular.  The  Delphi  belief  structure  is  psychologically 
held  together  by  the  cementing  influence  of  the  expert  halo  effect. 

A  tacit,  largely  unchallenged  assumption  of  Delphi  is  that  authen¬ 
tic  experts  do  in  fact  exist  for  predicting  the  extremely  complex  socio¬ 
economic-technological  events  so  common  in  Delphi  questionnaires. 

Closer  scrutiny  revealo  this  to  be  wishful  thinking.  Many  of  these 
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events  ere  initial  forays  Into  unknown  areas  requiring  unknown  skills, 
hence,  unknown  "experts."  Even  if  such  events  are  understood  to  some 
extent,  they  typically  presuppose  a  fantastic  array  of  real,  not 
shallow,  skills  in  diverse  and  far-ranging  fields  such  as  economics, 
public  policy,  esoteric  technologies,  Individual  and  group  psychology, 
law,  medicine,  etc.,  which  is  simply  beyond  the  ken  of  any  living  mor¬ 
tal.  When  we  match  predictions  of  complex  sets  of  social  events 
against  "experts,"  we  get  something  like  the  fabled  blind  men  examining 
the  Indian  elephant.  If  we  think  of  experts  as  idiots  savants,  we 
suddenly  avoid  the  trap  of  the  expert  halo  effect. 

Another  central  postulate  in  the  Delphi  epistemology  of  experts 
is  that  they  will  in  fact  provide  significantly  better  and  substan¬ 
tially  different  responses  than  nonexperts.  Practically  every  Delphi 
practitioner  asserts  that  Delphi  outputs  are  only  as  good  as  the  ex¬ 
pert  Inputs,  admonishing  us  with  the  CIGO  principle  (garbage  In/garbage 
out) . 

Suppose,  however,  that  it  can  be  proven  that  any  informed  group 
of  individuals  in  the  object  area  of  inquiry  can  provide  individual 
and  group  Delphi  opinions  essentially  indistinguishable  from  those  of 
the  experts.  It  would  follow,  then,  that  Delphi  results  merely  repre¬ 
sent  informed  opinion  rather  *han  expert  opinion . 

Personal  experience  with  graduate  student  predictors  brought  this 
potential  expert  fallacy  to  the  author's  attention.  In  connection  with 
a  graduate- level  course  on  computers  and  society,  the  author  asked  his 
students  to  give  their  independent  estimates  of  expected  order  of  oc¬ 
currence  of  each  of  the  events  in  automation  (computer  technology)  and 
general  scientific  advances  originally  Investigated  by  Gordon  and 
Helmer  in  their  1964  Delphi  study.  (See  Fig.  1  for  results  with  auto¬ 
mation  items.)  After  the  students  ranked  the  listed  events  they  were 
told  the  "true"  ranks  listed  by  the  experts  in  the  original  study,  and 
calculated  a  Spearman  rank  coefficient  (product-moment  correlation  of 
ranks).  This  provided  each  student  with  a  correlation  coefficient  com¬ 
paring  his  first-round  estimates  with  the  medians  of  the  "experts." 

Over  the  years,  we  have  consistently  found  median  rank-correlations  for 
classes  of  about  a  dozen  students  at  about  .70  for  both  areas  for 
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Fig.  1:  Delphi  Results  for  Progress  in  Automation 
(Adapted  from  Gordon  and  Helmer,  1964.) 
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firs  t-round  estimates.  These  results  are  roughly  equivalent  to  the 
upper  levels  of  reliability  for  Delphi  judgments  described  earlier 
from  Dalkey  (1969). 

In  a  nutshell,  "informed"  graduate  students  provided  essentially 
the  seme  forecasts  as  "experts."  The  students  did  have  the  advantage 
of  making  their  predictions  some  six  years  later  than  the  experts  in 
the  original  study,  items  were  not  presented  randomly,  there  was  no 
iteration  with  feedback,  and  standardized  Instructions  were  not  rigor¬ 
ously  observed  in  these  informal  classroom  exercises.  If  this  general 
equivalence  holds  under  controlled  experimental  conditions,  anyone 
with  some  professional  training  in  broad  target  fields  could  play  the 
Delphi  game  and  it  wouldn't  make  any  difference  in  the  results. 

The  tests  using  graduate  students  were  not  conducted  as  rigorous 
experiments  and  the  results  have  not  been  reported  in  the  literature. 

No  claims  are  made  for  the  validity  of  the  findings — they  are  presented 
here  to  point  up  a  central  hypothesis.  Some  critical  experimental 
studies  comparing  experts  with  less  informed  individuals  and  with  non¬ 
experts  have  been  performed  in  the  Delphi  literature  and  in  precursor 
studies.  This  is  a  central  empirical  question  that  can  be  very  easily 
tested. 

At  the  beginning  of  this  section,  the  studies  of  Cantril  (1938) 
and  McGregor  (1938)  were  cited.  In  these  studies,  the  expertness  of 
the  forecaster  was  shown  to  have  little  or  no  significance  in  the  de¬ 
termination  of  predictions  of  complex  social  events.  More  precisely, 
no  statistically  significant  differences  in  such  predictions  were  found 
between  students  and  teachers,  laymen  and  professionals,  in  tests  which 
involved  a  combined  respondent  sample  of  over  600  subjects.  Predic¬ 
tions  were  demonstrably  linked  to  values  and  attitudes  toward  the  sub¬ 
ject  matter. 

Kaplan,  Skogstad,  and  Glrshlck  (1950)  applied  a  forecasting  ques¬ 
tionnaire  on  152  social  and  technological  events  to  26  subjects  repre¬ 
senting  the  entire  spectrum  from  senior  professional  to  layman.  Part 
of  the  study  involved  administration  of  a  general  knowledge  paper-and- 
pencil  test  on  "Current  Social  Problems"  and  "Science."  The  better- 
informed  subjects  (upper  half)  performed  only  slightly  better  than  the 
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leas  Informed  subjects  (lower  half) — average  accuracy  scores  for  short¬ 
term  predictions  were  56  percent  and  50  percent,  respectively.  This 
result  Is  In  the  expected  direction,  but  is  not  statistically  signifi¬ 
cant  with  respect  to  a  test  for  the  mean  difference  between  proportions 
for  this  sample.  Further,  the  amount  of  the  difference,  as  indicated 
by  the  authors.  Is  not  substantial.  Thus,  these  pre-Delphi  studies 
indicate  that  expertise  maker  either  no  difference,  or  only  a  trivial 
difference,  in  forecasting  a  variety  of  social  and  technological 
events . 

Much  the  same  results  occur  with  Delphi  studies.  In  Campbell's 
doctoral  dissertation  on  forecasting  short-term  economic  indicators 
(1966),  level  of  expertise  was  tested  in  terms  of  self-confidence  rat¬ 
ings.  He  correlated  these  ratings  for  each  item  against  forecasting 
accuracy  and  found  the  results  did  not  differ  significantly  from  a 
median  correlation  of  zero.  Campbell  concluded  that  "Selecting  the 
most  self-confident  members  of  a  group,  based  on  the  five-point  or  the 
group  self-confidence  scales,  was  not  an  effective  means  of  identifying 
the  most  accurate  forecasters"  (p.  112). 

Campbell  had  additional  information  for  a  further  test  of  the  re¬ 
lation  of  expertise  to  accuracy  in  forecasting.  Of  the  two  seminar 
groups  tested,  one  group  was  older  and  more  experienced  in  professional 
economic  forecasting  than  the  other.  The  more  experienced  group  did 
obtain  accurate  median  forecasts  more  often  than  the  less  experienced 
group  in  a  paired-comparison  test,  but  the  results  were  not  statisti¬ 
cally  significant  for  Delphi  and  non-Delphi  groups  matched  against  each 
other  for  16  economic  indicators.  (Because  Campbell  did  .not  report 
statistical  comparisons,  the  author  applied  the  non-para£etrlc  sign 
test  uaed  by  Campbell  In  similar  comparisons  and  obtained  confirmation 
of  the  null  hypothesis  for  Delphi  and  non-Delphi  groups.)  The  pooled 
results  showed  20  more  eccurete  forecasts  for  the  more  expert  group, 

10  for  the  less  expert  group,  and  two  ties,  which  meets  a  10  percent 
level  of  significance. 

Dalkey  (1969),  also  using  self-confidence  ratings  of  expertness 
for  each  item,  was  able  to  compare  those  "more  expert"  against  those 
"less  expert"  for  almanac-types  questions.  "The  basic  hypothesis  being 
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tested  was  that  a  subgroup  of  bo  re  knowledgeable  individuals  could  be 
selected  in  terns  of  their  self-rating,  and  that  this  group  in  general 
would  be  more  accurate  than  the  total  group.  In  every  case  this  hy¬ 
pothesis  was  not  confirmed"  (p.  68). 

In  a  subsequent  almanac-item  study,  Dalkey,  Brown,  and  Cochran 
(1969)  did  find  that  "...  significant  Improvements  in  accuracy  of 
group  estimates  can  be  obtained  with  proper  use  of  self-ratings" 

(p.  v.)  Close  examination  of  "proper  use"  reveals  rather  arbitrary 
ex  pos t  facto  statistical  verifications  that  have  dubious  generality 
for  other  studies  (e.g.,  at  least  seven  subjects  in  high  and  low  sub¬ 
groups,  with  no  overlap  in  self-ratings  between  subgroups,  which  eli¬ 
minated  many  of  the  subgroups).  Such  arbitrary  ad  hoc  statistical 
procedures  capitalize  on  chance  fluctuations  in  the  experimental 
sample.  A  more  appropriate  statistic  would  include  all  data,  such  as 
a  correlation  coefficient  showing  both  the  statistical  significance 
and  strength  of  the  association  between  self-ratings  and  accuracy. 

Bedford's  study  (1972)  Is  probably  the  most  relevant  to  the  issue 
at  point — are  there  demonstrable  forecasting  differences  between  ex¬ 
perts  and  non-experts?  Bedford  matched  a  group  of  25  housewives 
against  a  group  of  26  experts  in  "communications,  consumer  behavior, 
sociology,  and  futurism  generally"  in  a  two-round  Delphi  study  on  "The 
Future  of  Communications  Services  in  the  Home."  Bedford  found,  using 
a  long  and  extensive  questionnaire,  that  "There  were  remarkably  few 
differences  between  the  experts  and  the  housewives  on  the  panel" 

(p.  1).  His  results  support  the  contention  thet  level  of  expertise 
oakes  little  difference  in  exploratory  socio-economic  forecasts. 

Similar  results  were  obtained  by  Reisman,  Mantel,  Dean,  and 
Elaenberg  (1969)  in  a  comparative  Delphi  study.  Evaluative  ratings 
of  laymen  correlated  highly  with  ratings  of  experts  for  250  social 
service  packages  handled  by  the  agencies  of  the  Jewish  Community 
Federation  of  Cleveland.  These  results  also  tend  to  support  the  hy¬ 
pothesis  that  opinions  for  evaluative  social  areas  of  Inquiry  tend  to 
be  Independent  of  level  of  expertise. 

What  is  the  box  score  for  the  null  hypothesis  that  there  are  no 
demonstrable  differences  between  predictions  of  experts  and  non-experts 


-40- 


for  socio-economic- tecnnological  events?  The  McGregor  (1938)  end 
Cantril  (1938)  studies  unequi vocally  indicate  that  such  differences  do 
not  exist  for  complex  social  events  impacting  on  personal  values.  The 
Bell  Canada  study  by  Bedford  (1972)  indicates  that  no  demonstrable  » 
differences  were  shown  between  housewives  and  experts  for  soclo- 
technological  developments.  Campbell's  analysis  of  self-confidence 
ratings  also  supports  the  null  hypothesis,  in  that  no  correlation  was 
obtained  with  accuracy  of  short-term  economic  forecasts.  Dalkey'a 
1969  study  showed  no  differences  in  almanac  items  estimates  with  re¬ 
spect  to  ratings  of  self-confidence.  Reisman  et  al.  (1969)  showed 
similar  responses  from  laymen  and  experts  in  evaluations  of  social 
services.  These  studies  collectively  indicate  that  it  doesn't  make 
any  difference  how  expert  the  respondent  is,  or  how  confident  he  feels 
about  his  opinion,  when  forecasting  or  estimating  a  vide  variety  of 
social,  economic,  and  technological  phenomena. 

Studies  that  show  some  differences  In  responses  between  different 
levels  of  expertise  are  marginal  at  best.  The  Kaplan,  Skogstad,  and 
Girshick  study  showed  a  statistically  non-significant  trend  in  the 
correct  direction  with  more  "knowledgeable"  subjects  contributing  more 
accurate  short-term  forecasts.  Campbell's  data  (1966)  also  showed  a 
statistically  non-significant  trend  in  the  expected  direction  with  his 
more  experienced  group  tending  to  give  more  accurate  forecasts  than  the 
less  experienced  group.  The  Oalkey,  Brown,  and  Cochran  study  (1969) 
showed  statistically  marginal  results  in  the  expected  direction  for 
self-confidence  ratings. 

If  Delphi  investigators  cannot  demonstrate  statistically  signifi¬ 
cant  and  substantial  differences  between  experts  and  non-experts,  then 
it  must  be  concluded  that  the  Delphi  emphasis  on  the  use  of  experts  is 
misplaced.  Available  experimental  data  indicate  that  this  conclusion 
is  probably  the  most  accurate  generalization  for  most  Delphi  applica¬ 
tions.  If  statistically  significant  but  lw-order  correlations  are 
found,  the  expert  concept  is  only  marginal,  and  virtually  worthless 
from  a  practical  point  of  view.  The  above  experimental  data  indicate 
that  this  might  be  the  case  in  a  small  proportion  of  well-defined  and 
highly  specialized  applications.  If  significant  and  substantial 
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dlfferences  are  founds  a  stronger  case  may  be  made  for  Delphi  expert 
opinion  for  the  target  area  of  Inquiry.  The  above  experimental  data 
offer  no  evidence  of  substantial  differences  between  experts  and  non¬ 
experts. 

Looking  back  at  the  central  issue  of  the  Delphi  concept  and  use 
of  experts  as  discussed  in  this  section,  we  find  the  following  short¬ 
comings  : 

o  The  concept  of  expert  is  virtually  meaningless  in  ex¬ 
periments  dealing  with  complex  social  phenomena. 

o  Sole  or  primary  reliance  on  expert  opinion  in  the 
social  sciences  has  long  been  discredited  and  now 
has  no  serious  advocates. 

o  Anonymous  panels  chosen  in  unspecified  ways  enhance 
the  possibilities  for  contaminated,  elitist  "expert" 
samples. 

o  There  exists  an  imeon trolled  and  unknown  expert  halo 
effect  in  Delphi  contributing  to  expert  oversell. 

Collective  expert  opinion  directly  reinforces  un- 

/ 

accountability  for  Delphi  results  for  all  concerned — 
the  director,  panelists,  and  users. 

o  Experts  and  non-experts  consistently  give  indistin¬ 
guishable  responses  in  forecasting  or  evaluating 
social  phenomena  impacting  on  common  values  and  at¬ 
titudes  . 

o  There  is  no  explicit  matching  of  skills  required  by 
Delphi  questions  against  objectively  measureable 
skills  of  the  panelists. 

The  difficulties  associated  with  the  Delphi  concept  of  "expert" 
does  not  and  should  not  imply  that  all  and  any  use  of  experts  is  neces¬ 
sarily  bankrupt.  The  originators  of  Delphi  should  be  credited  with 
clearly  sensing  and  trying  to  respond  to  strong  social  demand  for  ex¬ 
ploiting  expert  opinion  more  effectively.  For  example,  in  a  survey  of 
65  corporations,  Hayden  (1970)  found  that  69  percent  used  diverse 
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expert  panel  consensus  techniques,  end  of  these,  26  percent  used  Delphi. 
This  example  is  probably  Indicative  of  the  widespread  informal  end 
formal  use  of  "experts"  throughout  society.  The  proper  use  of  expert 
talent  remains  a  major  problem  of  our  time.  We  know  precious  little 
about  the  dynamics,  the  use,  and  the  abuse  of  experts  in  our  society. 
Substantive  treatment  of  this  problem,  however,  is  beyond  the  scope 
of  this  analysis. 

4.2.  Face-to-Face  Confrontation  Versus  Private  Opinion 

Much  of  the  popularity  and  acceptance  of  Delphi  rests  on  the  claim 
of  the  superiority  of  group  over  individual  opinions,  and  the  prefer¬ 
ability  of  private  opinion  over  face-to-face  confrontation.  Martino 
(1972),  for  example,  flatly  asserts  "It  should  be  remembered  that 
Delphi  represents  a  distinct  improvement  over  either  individual  experts 
or  face-to-face  panels"  (p.  27). 

Democratic  process  rests  on  the  secret  ballot  where  voting  is  per¬ 
formed  in  private.  Group  opinion  is  a  time-honored  corrective  against 
individual  excesses.  And  how  many  of  us  have  either  bean  bullied  in 
heated  group  exchanges  or  have  bullied  others  when  we  had  the  opportu¬ 
nity?  Besides,  who  wants  to  take  the  time  and  effort  to  travel  to  a 
meeting  and  listen  to  every  panelist  defend  his  expertise  to  the  rest 
of  the  group?  A  quick  and  incisive  statement  of  the  issues  on  paper 
and  an  equally  quick  indication  of  individual  opinion,  also  on  paper 
and  in  the  familiar  privacy  of  your  c*m  office,  as  advocated  by  Delphi, 
has  almost  irresistible  practical  appeal  as  a  sensible  and  cost- 
effective  solution  to  the  problem  of  sampling  expert  opinion. 

On  the  other  hand,  each  of  us  can  probably  recount  numerous  ex¬ 
amples  where  individuals  were  more  effective  than  groups  in  arriving 
at  informed  opinion;  where  confrontation  clarified  the  issues  and  made 
honest  communication  possible;  where  Introversion  and  isolation  lad  to 
unfortunate  aberrations  of  opinion  and  outlook. 

The  experimental  data  comparing  individual  and  group  performance 
offer  no  convincing  conclusions  on  either  side  of  these  broad  lasuas, 
although  the  literature  extends  over  many  decades.  After  reviewing 
the  early  literature  in  this  area  (1920-1957),  Lorge,  Fox,  Davits,  and 
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B rennet  (1958)  indicate  that  superiority  of  the  group  or  the  individual 
is  relative  to  stipulated  experimental  tasks  and  conditions,  varies 
enormously  with  individual  differences,  and  Is  shot  through  with  meth¬ 
odological  difficulties  in  generalizing  from  experimental  to  real- 
world  situations.  In  a  more  recent  review  of  the  experimental  litera¬ 
ture,  Haler  (1967)  concludes  that  the  comparative  effectiveness  of  in¬ 
dividuals  versus  groups  varies  widely  and  depends  upon  the  tradeoff 
of  the  assets  and  liabilities  of  both  approaches  in  the  unique  applied 
setting.  He  emphasizes  the  crucial  role  played  by  experienced  group 
leaders  acting  as  neutral  facilitators  in  achieving  successful  group 
outcomes . 

If  we  look  for  Delphi  studies  comparing  groups  and  individuals  we 
find  a  near  vacuum.  Dalkay  (1969)  compared  face-to-face  with  anonymous 
Delphi  Interaction  for  the  almanac-type  items  mentioned  earlier.  He 
found  a  tendency  toward  more  accurate  opinion  in  the  anonymous  setting, 
a  statistically  non-significant  tendency.  Dalkey's  procedure  involved 
picking  group  "leaders"  randomly,  which  flies  in  the  face  of  effective 
group  procedure  and  effectively  stacks  the  odds  against  successful 
group  interaction.  Farqunar  (1970)  compared  group  versus  anonymous 
Delphi  interaction  for  a  complex  software  estimation  task  and  consis¬ 
tently  obtained  substantially  better  results  in  the  face-to-face  group. 

Campbell's  dissertation  (1966)  is  frequently  cited  by  Delphi  pro¬ 
ponents  as  definitive  evidence  of  the  superiority  of  Delphi  group 
opinion  compared  with  face-to-face  confrontation  in  traditional  expert 
panels.  Campbell  worked  out  s  careful  experimental  design  as  far  as 
subject  sampling  is  concerned,  randomly  assigning  graduate  student 
participants  to  experimental  Delphi  panels  and  control  confrontation 
groups  (which  he  called  "uncontrolled-interaction  groups").  His  cri¬ 
terion  measure  consisted  of  accuracy  in  forecasting  16  short-term  sta¬ 
tistical  economic  indicators;  a  flaw  in  this  part  of  his  study  is  that 
these  16  measures  are  only  partially  independent,  which  vitiates  the 
integrity  of  statistical  tests  based  on  assumptions  of  independence. 
Campbell  used  nonparauetrlc  statistics  in  comparing  median  forecasting 
performance  cf  his  experimental  (Delphi)  and  control  groups  (confronta¬ 
tion)  ,  and  apparently  demonstrated  statistically  significantly  better 
forecasting  in  his  two  matched  Delphi  groups. 


His  conclusion,  however,  is  based  on  a  straw-man  type  of  compari¬ 
son,  similar  in  certain  respects  to  the  token  conventional  group  struc¬ 
ture  used  by  Dalkey  <1969),  mentioned  earlier.  Campbell's  control 
groups  were  leaderless,  and  remained  leaderless,  which  undoubtedly  led 
to  considerable  floundering  and  non-miBsion  posturing  and  competition. 
The  simple  institution  of  an  elected  chairman  to  organise  each  group, 
who  would  identify  with  the  problem,  as  occurs  in  conventional  commit¬ 
tees,  might  have  altered  results  substantially.  The  confrontation 
groups  were  force-fitted  into  a  Delphi-type  format  to  make  quantitative 
forecasts  more  directly  comparable.  For  example,  meetings  were  kept 
within  fixed  periods  of  time,  whether  or  not  the  group  wanted  such  a 
procedure,  with  one  meeting  corresponding  to  each  round  of  the  Delphi 
panels;  discussion  of  each  economic  Indicator  was  also  pegged  to  a 
fixed  period  of  time,  regardless  of  success  or  failure  in  achieving 
closure  or  consensus;  and  each  meeting  required  open  individual  polling 
for  statistical  comparability  of  estimates  between  experimental  and 
control  groups,  whether  or  not  the  group  wanted  to  follow  such  a  pro¬ 
cedure.  These  procrustean  constraints  break  most  of  the  rules  for 
professional  or  enlightened  group  problem  solving.  The  oppressiveness 
of  these  artificial  confrontation  groups  may  have  undermined  group 
motivation  and  morale  to  the  point  where  the  meetings  became  counter¬ 
productive,  and  the  comparison  spurious.  Accordingly,  Campbell's 
study  can  not  be  viewed  as  a  serious  comparison  of  the  effectiveness 
of  Delphi  and  conventional  panels  for  his  criterion  measures- 

The  results  raise  additional  methodological  problems.  Campbell 
did  not  compare  the  forecasting  results  of  both  types  of  groups  against 
trend  extrapolations  of  his  selected  economic  time  series,  even  though 
these  series  were  available  on  a  quarterly  basis.  It  may  be  that 
simple  arithmetic  extrapolation  (as  mentioned  earlier  in  connection 
with  Zamowitz's  critical  review  of  expert  economic  forecasting),  or 
perhaps  more  sophisticated  multiple  regression  analyses,  might  provide 
results  as  good  or  better  than  those  obtained  with  expert  groups. 
Finally,  quarterly  forecasting  is  hardly  a  criterion  vehicle  for  an 
expert  panel  when  reliable  and  extensive  baseline  statistical  data  are 
available  for  fine-grain,  short-term  trend  forecasting.  Any  general¬ 
ization  from  such  results  would  have  to  be  limited  to  very  short-range 
forecasting. 
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The  allaged  superiority  of  anonymous  Delphi  opinion  over  face-to- 
face  opinion!  and  its  converse!  arc  provable  general  propositions. 
They  can  not  be  proved  or  disproved!  in  general,  because  the  proposi¬ 
tions  are  amorphous  stereotypes  and  are  not  amenable  to  scientific 
testing  unless  they  are  operationally  defined.  Once  such  definition 
is  applied  to  limited  concrete  situations,  one  approach  may  prove  more 
effective  than  another,  both  approaches  may  be  more  powerful  than 
either  alone,  or  the  two  approaches  may  be  so  close  as  to  not  make  much 
practical  difference.  Investigators  should  be  more  interested  in  a 
flexible  eclectic  approach  that  freely  capitalizes  on  the  best  of  both 
worlds  than  in  identifying  with  a  ritualized  approach  on  either  side. 

In  any  case,  the  Delphi  claim  that  pooled  group  "expert"  opinion  is 
more  effective  than  individual  opinion,  and  that  anonymous  interaction 
Is  more  effective  than  direct  confrontation,  cannot  be  sustained. 

4.3.  Delphi  Consensus 

The  goal  of  the  Delphi  procedure  la  to  arrive  at  a  meeting  of  the 
minds,  consensus  among  the  exparts.  The  position  taken  here  Is  that 
the  Delphi  procedure  arrives  at  such  a  consensus  by  feeding  back  the 
"correct"  answer,  by  rewarding  conformity  and  effectively  penalizing 
individuality,  and  by  proffering  non-independent  iterative  results  as 
authentic  expert  consensus.  Authentic  consensus  refers  to  group  agree¬ 
ment  reached  as  a  result  of  mutual  education  through  increased  informa¬ 
tion  and  the  adversary  process,  which  leads  to  improved  understanding 
and  insight  into  the  issues;  it  does  not  refer  to  changes  of  opinion 
associated  primarily  or  exclusively  with  bandwagon  statistical  feed¬ 
back. 

It  was  stated  earlier,  In  connection  with  the  APA  professional 
standards  for  soliciting  judgment ,  expert  or  otherwise,  with  a  stan¬ 
dardised  instrument,  that  the  judgment  should  be  independent.  The 
first  Delphi  round  represents  independent  opinion,  whereas  succeeding 
rounds  are  strictly  correlated.  First-round  results  of  "experts"  may 
contain  a  range  of  responses  up  to  four  orders  of  magnitude  for  some 
types  of  quantitative  estimates  (sec  Dalkey,  1969,  and  Baran,  1971), 
which  are  hardly  publishable  as  "consensus."  Raw-score  frequency 
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distributions  are  so  highly  skewed  chat  logarithmic  trans format ions 
are  often  required  to  approximate  normal  distributions.  Perhaps  this 
is  why  most  Delphi  investigators  do  not  report  first-round  dispersions. 
(Borko,  1970,  provides  an  exception  to  this  rule.) 

An  example  of  the  logarithmic-range  of  first-round  dispersions 
for  some  types  of  Delphi  estimates  is  provided  by  Baran  (1971)  in  an 
illustrative  appendix  of  his  report  for  "Cashless-Society  Transactions." 
Tills  item  refers  to  cost  and  marketing  estimates  of  hard-copy  recording 
of  financial  transactions  with  update  balance  in  computer  memory.  The 
first  round  showed  a  range  of  $.01  to  $100  for  average  dollar  value 
.-f  a  transaction  (10,000:1),  5  percent  to  90  percent  market  penetration 
five  years  after  mass  introduction  of  this  service,  and  a  range  of  0 
percent  to  100  percent  for  percentage  of  this  service  that  home  sub¬ 
scribers  would  be  expected  to  pay.  The  Inkblot  nature  of  such  future 
projections  speaks  for  itself. 

Now,  in  succeeding  rounds,  do  the  panelists  really  think  through 
their  positions  and  work  toward  authentic  consistency  of  opinion,  or 
are  they  effectively  pressured  into  conformity?  Dalkey  (1969)  has  in¬ 
dicated  that  statistical  feedback  alone  (group  medians  for  each  item) 
is  as  effective  in  obtaining  consensus  as  statistical  feedback  with 
adversary  rationale  for  responses.  Once  the  panelist  knows  the  median 
for  a  problematic  item,  he  has  in  a  very  real  sense  been  given  the 
"correct"  answer  to  the  item.  Panelists  are  quite  aw are  that  median 
responses  (or  some  other  measure  of  central  tendency)  are  offered  as 
best  estimates  for  questionnaire  items  in  the  final  results. 

Social  psychologists  have  long  been  aware  of  powerful  tendencies 
for  individuals  to  conform  to  group  opinion  in  relatively  wstructured 
situations,  particularly  if  the  motivation  level  is  not  high  (Stogdill, 
1959,  and  Berelson  and  Steiner,  1964).  The  "autokinetic"  effect  is  a 
striking  example  of  this  tendency  (Sherif,  1936).  Place  an  Individual 
or  a  group  of  people  in  a  completely  darkened  room  with  a  single,  flxad 
point  of  light.  The  light  will  appear  to  drift  randomly  with  a  dis¬ 
placement  as  high  as  20  degrees,  because  of  the  absence  of  a  visual 
frame  of  reference.  (Astronomers  were  the  first  to  notlee  and  study 
the  autokinetic  effect.)  Ask  the  subjects  in  such  a  room  to  estimate 
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the  direction  and  amount  oC  perceived  movement  of  the  light.  Initial 
random  Judgments  toon  converge  closely  around  the  group  norm  after  a 
few  rounds  of  group  opinion.  Group  suggestion  provides  the  "correct'' 
answer  to  an  inherently  ambiguous  situation.  Consensus  is  specious. 

Figure  2  shows  sane  of  Sheri f's  experimental  results  with  the 
autok luetic  effect.  The  first  session  involved  individuals  alone  re¬ 
porting  observed  deviations  of  the  pinpoint  of  light  in  a  completely 
darkened  room.  The  ordinates  in  Fig.  2  represent  median  deviations 
ic  inches,  the  abcissas  represent  successive  sessions  (equivalent  to 
Ilelphi  roun's).  The  second,  third,  and  fourth  rounds  were  group  ses- 
stems  where  each  individual  hed  an  opportunity  to  hear  the  deviations 
reported  by  others.  Note  that  the  individual  median  deviations  rapidly 
converge  to  a  group  norm  by  the  fourth  round  for  groups  of  two  or  three 
subjects. 

The  analogy  with  Delphi  is  startling-  Convergence  of  medians  is 
greatest  with  initial  feedback  of  group  opinion,  and  is  effectively 
achieved  in  three  to  four  rounds.  Delphi  investigators  typically 
reach  the  point  of  diminishing  returns  at  about  three  or  four  roundc 
as  far  as  measurable  convergence  of  opinion  is  concerned.  When  we 
couple  Sherif's  results  with  Dalkey's  assertion  that  ntatistlcal  re¬ 
sponse  alone  is  the  most  effective  way  to  achieve  couseasus  (without 
verbal  feedback)  we  have  the  artifact  of  autokinetlc  consensus  (group 
suggestion)  explaining  Delphi  consensus.  Sherif  ran  many  variations 
of  the  autokln>  Vic  effect  demonstrating  easily  manipulated  shifts  in 
subjects'  opinions  in  any  desired  direction  by  suggestion  from  the 
experimenter  (e.g.,  "you  are  underestimating  light  movement")  or  from 
other  authority  figures,  such  as  group  leaders.  The  uncontrolled, 
arbitrary  introduction  of  selected  verbal  feedback  by  the  Delphi  di¬ 
rector  can  with  corresponding  ease  shift  ootnlansj in  desired  direc¬ 
tions.  \ 

The  Delphi  technique  thus  deliberately  manipulates  responses  to¬ 
ward  minimum  dispersion  of  opinion  in  the  name  of  concensus.  The  pre¬ 
sentation  of  median  opinions  (after  the  first  round)  anch<5^  coercion 
toward  conformity  are  reassuringly  represented  to  all  as  reasoned 


Subject  1 
Subject  2 
Subject  3 
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consensus.  By  the  time  the  third  or  fourth  round  occurs,  the  holdout 
individualist  responses  pose  the  threat  of  yet  another  cedious  run 
through  the  same  items,  and  even  die-hards  are  inclined  to  yield  ro 
save  everyone  the  dreary  routine,  of  another  round.  Martino  (1972) 
recommends,  ".  .  .  in  many  cases,  there  is  no  advantage  in  going  be¬ 
yond  two  rounds"  (p.  27). 

In  passing,  it  should  be  noted  that  the  term  "panelist"  is  a  mis¬ 
nomer  in  the  Delphi  context.  Panelists  usually  communicate  directly 
and  exchange  opinion  with  each  other,  primarily  in  a  face-to-face  set¬ 
ting.  With  Delphi,  we  have  respondent,  not  panelists,  because  com¬ 
munication  is  strictly  with  the  questionnaire,  not  with  other  people. 
Moreover,  all  responses  are  filtered  through  the  intermediary  of  the 
Delphi  director  or  his  representative  before  reaching  anyone  else. 

There  is  no  Interactive  discourse  denerving  of  the  name  "panel"  in 
Delphi  procedures.  Respondents  really  represent  a  non-communicating 
non-group.  Jinked  primarily  by  remote  statistical  feedback. 

Delphi  consensus  is  suspect  from  still  another  viewpoint.  The 
first-round  items  are  quite  different  vhe.u  they  are  accompunled  by 
statistical  and  verbal  feedback  provided  by  the  director  in  succeeding 
rounds.  Once  the  Information  accompanying  an  item  is  altered,  it  is 
literally  a  different  item.  Just  as  minor  rewording  can  change  a 
questionnaire  item  enormously,  so  does  Delphi  "feedback"  change  the 
item  in  uncontrolled  and  unknown  ways.  How  can  medians  and  disper¬ 
sions  be  compared,  and  consensus  claimed,  if  items  are  noncomparable 
from  round  to  round? 

The  social  implj cations  of  specious  consensus  are  enormous. 
Variations  of  similar  iterative  query  techniques,  with  conformist- 
reinforced  feedback,  provide  almost  unlimited  possibilities  for  shap¬ 
ing  and  manipulating  public  opinion  via  the  interactive  communications 
media  of  the  future. 

4.4.  Delphi  Questionnaire  Items 

The  basic  criticism  leveled  against  Delphi  questionnaire  items  is 
that  they  are,  by  and  large,  unavoidably  amorphous.  More  specifically. 
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complex  future  events  (end  value  Judgments)  do  not  lend  themselves  to 
clear  and  unaablguous  description  In  typical  one-sentence  Delphi  ques¬ 
tionnaire  format  (e.g.,  see  the  automation  Items  from  the  Got  don -He  laker 
study  in  Fig.  1).  Instead  we  find  vague,  generalised  descriptions  of 
future  events,  permitting  the  respondent  to  project  any  one  of  a  large 
number  of  possible  scenarios  as  his  particular  interpretation  of  that 
event.  Delphi  asks  panelists  about  e/ent-stereotypes,  and  panelists 
respond  with  stereotype  estimates.  Delphi  verbal  responses,  when  they 
occur,  are  typically  vague  and  sweeping  descriptions,  slogans,  or  sim¬ 
plistic  statements. 

More  thoughtful  and  careful  Delphi  Investigators  attempt  to  qual¬ 
ify  forecasts  by  identifying  percentages  of  specific  respondent  popu¬ 
lations  and  by  associating  probability  estimates  with  predlcltons. 

Such  attempts,  although  in  the  right  direction,  arc  no  substitute  for 
precisely  defined,  detailed  scenarios  for  each  Item  where  a  host  of 
assumptions  specifying  the  "event"  are  made  explicit.  The  question¬ 
naire  format  does  not  lend  itself  to  such  presentation. 

For  example,  the  Delphi  Inquiry  might  be  concerned,  as  in  Baran's 
study  (1971),  with  the  "Potential  Market  Demand  for  Two-Way  Informa¬ 
tion  Service  to  the  Home."  Baran  had  to  leave  vast  areas  unspecified 
in  asking  panelists  when  such  services  were  likely  to  be  available 
and  how  much  they  would  cost  the  consumer.  These  wspecified  areas 
Included  the  configuration  of  hardware,  softvere,  and  communications; 
the  nature  of  federal,  state,  and  local  regulation  of  such  mass  com¬ 
puter  services;  the  mix  of  public  and  private  support  of  the  Informa¬ 
tion  services  considered;  very  brief  general  descriptions  of  the  30 
Information  services  (typically  one  paragraph);  no  indication  of  how 
the  public  will  be  taught  to  use  such  services;  and  many  other  socio¬ 
economic-technological  areas  Impacting  directly  on  theae  services. 
Baran's  study  is  probably  one  of  the  best  available  in  the  Delphi 
literature,  featuring  extensive  use  of  computer  support,  and  a  ratio¬ 
nal  quantitative  and  probabilistic  cost  format  for  couching  question¬ 
naire  items.  But  even  with  all  theaa  precautions,  which  are  consider¬ 
ably  more  than  are  encountered  la  the  typical  Delphi  study,  the  items 
incorporate  vast  areas  of  ambiguity  and  represent  an  array  of  possible 
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speciflc  events  "fitting"  into  eech  item.  (Recell  the  "cashless-society 
transaction"  item,  cited  previously.)  As  presently  practiced,  Delphi 
4s— in  many  respects— a  psychological  projective  technique  for  future 
inkblots . 

In  hl«  recent  experimentation  with  Delphi  procedures  in  the  field 
of  drug  abuse,  Thompson  (1973)  underscores  the  top-priority  need  for 
extensive  pre-testing,  and  the  great  difficulties  encountered  in  de¬ 
veloping  reliable  and  useful  Delphi  questionnaires. 

"The  loose  challenging  aspect  of  future  applications  of 
Delphi  techniques  to  the  drug  field  will  almost  certainly 
be  the  design  of  a  cohesive  set  of  questionnaire  items 
that  are  both  well-posed  and  useful  to  the  decisionmaker. 

On  the  one  hand,  it  became  apparent  during  the  study  that 
developing  concise  questions  which  will  be  given  similar 
interpretations  by  all  respondents  will  inevitably  in¬ 
volve  substantial  pre-testing.  The  usual  difficulties 
in  questionnaire  design  are  compounded  in  the  context 
of  drug  abuse  by  disagreement  over  underlying  assump¬ 
tions,  and  by  the  absence  of  an  agreed-upon  vocabulary." 

The  psychological  literature  on  attitude  and  opinion  testing  has 
described  an  instructive  historical  process  that  appears  to  have  gone 
unnoticed  by  the  Delphi  cooaunlty.  After  an  initial  era  of  free¬ 
wheeling,  broad-gauged  questionnaires  on  attitudes,  covering  almost 
anything  of  interest,  the  evolutionary  trend  has  been  toward  highly 
specialised  attitude  and  opinion  Instruments  concerned  with  investiga¬ 
tion  of  specific  Issues  In  depth  (e.g.,  Anastasl,  1968).  In  the  Delphi 
context,  this  means  that  single  items  are  often  of  sufficient  complex¬ 
ity  to  warrant  construction  of  a  complete  questionnaire  dedicated  ex¬ 
clusively  to  that  itea,  exploring  aajor  lap licat ions  and  aspects,  to 
better  reveal  the  constellation  of  opinions  to  which  it  gives  rise. 

This  permits  the  development  snd  test  of  theory  to  explain  and  enhance 
understanding  of  the  item  or  issue  in  question. 

4.5.  Delphi  Responses 

If  Delphi  questions  are  ablguous,  then  Delphi  responses  are  also 
ambiguous ,  The  structure  and  dynamics  of  Delphi  responses  contribute 
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to  compounding  the  ambiguity.  This  is  due  to  several  factors.  Among 
the  most  important  is  the  pitfall  of  inviting  snap  answers  to  amorphous 
questions . 

Delphi  investigators  rarely  analyze  and  report  the  effort  panelists 
put  into  responding  to  their  questionnaires.  Solicitation  requests, 
and  Instructions  accompanying  Delphi  questionnaires,  typically  assure 
the  panelist  that  the  forms  can  be  quickly  and  easily  filled  out.  If 
not,  the  investigator  runs  the  risk  of  massive  dropout  rates,  as  oc¬ 
curred  with  a  19  percent  first-round  response  in  Kochman's  study  (1968). 
Assurances  are  often  provided  that  forms  should  not  normally  require 
more  than  about  an  hour  of  the  panelist's  time  for  each  round.  Martino 
(1972),  for  example,  recommends  an  upper  limit  of  25  items  for  Delphi 
ques  tionnaires . 

In  the  absence  of  information  on  panelist  effort,  the  author  timed 
his  own  responses  for  two  Delphi  studies  in  which  he  was  a  panelist. 

The  results  showed  great  variation  from  item  to  item,  with  an  average 
of  one  minute  per  item  where  few  comments  were  written,  to  an  average 
of  about  two  minutes  per  item  for  heavily  annotated  justifications  of 
responses.  The  typical  sequence  Would  be  to  read  the  item;  think 
quickly  about  key  critical  factors  influencing  the  forecast;  peg  the 
crucial  factor,  if  any,  or  fall  back  on  a  general  stereotype.  If  avail¬ 
able;  get  a  crude  estimate  of  its  occurrence;  and  fit  the  gross  esti¬ 
mate  into  the  questionnaire  scale.  The  average  one  minute  pass  per 
item  was  armchair,  top-of- the-head  opinion,  for  strictly  ball-park 
estimates.  The  average  two-minute-per-item  session  Involved  spending 
almost  three  hours  on  a  lengthy  form,  with  many  annotations,  which 
was  as  much  as  the  author  was  willing  to  contribute.  Although  this 
experience  is  not  necessarily  representative,  it  is  difficult  to  con¬ 
ceive  average  speeds  very  much  faster  than  a  fraction  of  one  minute 
per  item  or,  at  the  other  extreme,  Delphi  questionnaires  taking  more 
than  half  a  day  of  the  respondent's  time  for  a  single  round— even  if 
the  data  are  collected  in  the  costly  form  of  a  personal  Interview. 

The  author  is  participating  in  a  Delphi  study  being  conducted  by 
Bell  Canada  to  assess  future  home  communication  service  trends.  Re¬ 
sponse  times  for  the  first  round  have  been  carefully  recorded.  The 
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Deiphi  director  Indicated  in  the  instructions  chat  the  207-item  ques¬ 
tionnaire  should  not  require  more  than  an  hour  to  answer.  The  author 
took  65  minutes.  The  average  time  per  question  was  19  seconds.  The 
range  of  response  times,  for  different  groups  of  items  (usually  one 
page  per  group),  varied  from  40  seconds  per  question  Co  9  seconds. 

The  pacing  mechanism  was  "fastest  possible  reading  speed  for  compre¬ 
hension  and  Instant  response."  The  author  indicated  to  the  Delphi 
director  that  he  had  no  confidence  in  such  free-association  Judgments. 

An  "analysis"  averaging  one  minute  or  less  for  complex  forecasts 
is  merely  a  snap  judgment,  experts  notwithstanding.  The  results  are 
free-association  attitudes  toward  the  future,  not  analyses  of  future 
events.  We  also  tend  to  get  order-of-magnitude  responses,  particularly 
for  quantitative  data.  As  mentioned  earlier,  this  Is  particularly  ap¬ 
parent  In  first- round  results. 

Responses  tend  to  represent  stereotyped  thinking,  as  Illustrated 
by  the  follcx/ing  comments  taken  from  the  Nanus,  Wooten,  Borko  study 
(1973):  "this  technology  la  essentially  here  already,  so  I'll  fore¬ 
cast  early"  or  "utopian  dreamwork,  so  I'll  forecast  never,"  or  "costs 
are  much  to  high — appear  later,"  or  "no  one  cares,  the  public  won't 
buy  it,"  or  "this  is  a  trivial  advance,"  or  "this  will  kill  scienti¬ 
fic  progress,"  or  "people  will  rebel  against  this  invasion  of  privacy." 
This  is  not  to  deprecate  the  talent  and  experience  of  experts,  but 
most  human  beings,  when  placed  in  a  situation  where  they  are  regarded 
as  experts,  accountable  to  no  one,  and  expected  to  provide  quick 
answers  to  complicated  questions,  are  quite  likely  to  lean  very  hard 
on  stereotypes. 

The  hypothesis  has  been  advanced  in  various  contexts  in  previous 
sections  that  Delphi  forecasting  is  a  form  of  psychological  projection 
of  Inkblots  of  the  future.  Anyone  familiar  with  psychological  pro¬ 
jective  techniques,  such  as  the  Rorschach  Inkblot  test  and  the  Thematic 
Apperception  Test,  will  appreciate  the  fundamental  basis  of  such  tech¬ 
niques — there  are  as  many  "correct”  answers  as  there  are  respondents. 
The  respondent  projects  his  own  emotions,  needs,  attitudes,  imagina¬ 
tion,  experience,  stereotypes,  and  personal  problems  into  the  amor¬ 
phous  stimulus  situation,  modulated  by  distinguishable  cultural  factors 
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related  to  age  and  education  (Sackman,  1952).  We  saw  that  group  con¬ 
formity  factors  prevailed  in  the  autokinetic  aituatlon  studied  by  Sherlf 
(1936).  It  has  been  pointed  out  earlier  that  the  typical  single- 
sentence  questionnaire  format  for  Delphi  is  such  an  unstructured  stim¬ 
ulus  that  it  amounts  to  an  Inkblot  scenario  of  the  future.  We  have 
noted  that  one  minute  per  response  is  typical  of  Delphi  exercises;  it 
is  also  characteristic  of  psychological  Inkblot  tests  where  subjects 
are  urged  to  free-associate  to  amorphous  stimuli.  Rorschach  investi¬ 
gators  have  collected  thousands  of  responses  to  the  standardised  set 
of  ten  Inkblots  (originated  by  Hermann  Rorschach),  and  have  tallied 
responses  and  published  statistical  norms  of  popular  and  unusual  re¬ 
sponses.  They  do  not  assert  that  the  most  popular  responses  (e.g., 
butterfly,  dancing  girls)  are  "true"  or  "accurate"  responses.  By  the 
same  token,  Delphi  investigators  have  no  basis  for  equating  popularity 
with  validity  for  their  "inkblot"  results. 

Delphi  proponents  object  to  this  characterization  and  insist  that 
the  statement  of  Delphi  questions  in  objective,  quantitative  format 
yields  objective,  quantitative  results,  not  amorphous  personality 
projections  on  arbitrary  inkblots.  We  have  already  cited  order  of 
magnitude,  log-normal  dispersions  possible  for  first-round  quantitative 
estimates.  At  this  point,  additional  experimental  evidence  as  to  the 
underlying  dynamics  of  such  dispersion  is  presented  In  support  of  the 
hypothesis  that  Delphi  forecasts  sre  often  no  more  than  "Inkblot"  pro¬ 
jections  of  the  future. 

McGregor  conducted  a  large-scale  study  (1938)  of  psychological 
determinants  of  individual  predictions  of  social  events.  One  part  of 
his  study  was  concerned  with  the  impact  of  the  type  of  information 
given  to  respondents  when  thsy  were  asked  to  make  their  forecasts. 

Table  2,  reproduced  from  his  study,  shows  results  obtained  under  three 
conditions  in  response  to  the  requeat  to  estimate  the  site  of  the  Com¬ 
munist  Party  in  the  United  States  for  tha  next  yeer  (1936  at  the  time 
of  the  etudy) .  The  three  conditions  include  1)  no  information,  2) 
correct  information  (e.g.,  35,100  members  in  1935  with  official  figures 
for  prior  years,  and  3)  incorrect  information  where  tha  true  figures 
were  multiplied  by  five.  There  were  two  groups  of  subjects,  246  in  tha 
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first  and  376  In  Che  second.  The  first  group  effectively  went  through 
two  rounds  of  this  question.  Initially  with  no  information  (first  coluan 
of  results  in  Table  2) ,  and  later  with  Incorrect  information  (third 
column  in  Table  2). 

The  data  in  Table  2  reveal  several  notable  results.  First,  the 
estimates  of  the  uninformed  group  were  coo  high  by  an  order  of  magni¬ 
tude  (first  column  showing  an  interquartile  range  in  the  hundreds  of 
thousands).  Popularity  had  no  relation  to  accuracy.  Note  the  tendency 
toward  order-of-magnitude  clusters  at  tens  of  thousands,  hundreds  of 
thousands,  and  millions.  With  accurate  baseline  statistics,  as  ex¬ 
pected,  the  forecasts  in  the  second  column  of  Table  2  were  less  vari¬ 
able  and  far  more  accurate,  more  like  simple  short-term  trend  extra¬ 
polation.  The  third  column  in  Table  2,  roughly  analogous  to  a  second- 
round  Delphi  with  feedback,  shows  haw  easy  it  is  to  manipulate  quanti¬ 
tative  individual  and  group  opinion  to  cluster  closely  around  erroneous 
or  misleading  data  if  the  situation  is  sufficiently  unstructured.  The 
point  of  this  example  is  that  the  inkblot  hypothesis  applies  to  quan¬ 
titative  as  well  as  qualitative  data  for  unstructured  situations  such 
as  quantitative  Delphi  forecasts  of  complex  social  phenomena. 

We  have  already  discussed  the  contamination  of  opinion  with  "feed¬ 
back'  in  second  and  successive  rounds,  and  we  need  not  dwell  any  longer 
on  the  well-established  finding  that  individuals  tend  to  shift  their 
expectations  to  conform  to  overt  group  norms,  such  as  a  Delphi  median 
issuing  from  experts.  The  iterated  expert  response  to  each  Delphi 
item  is  thus  built  on  snap  judgment  on  the  first  round,  followed  by 
various  forms  of  overt  and  covert  conformist  pressure  in  succeeding 
rounds . 

4.6.  Delphi  Results 

Delphi  group  results  are  merely  collections  of  results  for  indi¬ 
vidual  questionnaire  items.  The  items  are  rarely  linked  together  with 
theoretical  or  systematic  constructs;  this  potpourri  contributes  to  a 
mixed  bag  of  findings.  As  mentioned  earlier,  item  reliability  and 
item  validity  are  typically  Ignored,  making  it  easier  for  the  unin¬ 
formed  user  to  accept  results  at  face  value.  Standard  errors  of 
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Table  2 

ESTIMATES  OF  THE  MEMBERSHIP  OF  THE  COMMUNIST  PARTY 
OF  THE  UNITED  STATES  UNDER  THREE  CONDITIONS* 


Inter-Quartile  Range 

Without 

Knowledge 

N  -  246 

100,000  to 
1,000,000 

With 

Correct 

Knowledge 

N  -  376 

33,000  to 
38,000 

With 

Incorrect 

Knowledge 

N  -  246 

160,000  to 
180,000 

Percent  predicting 
between  30,000  and 
40,000 

5 

83 

0.8 

Percent  predicting 
50,000  or  less 

21 

97 

2 

Percent  predicting 
between  150,003 
and  200,000 

7 

0.3 

76 

Percent  predicting 
1,000,000  or  more 

30 

0 

0 

Mean  prediction 

?b 

35,100 

172,000 

SOURCE:  Reproduced  free  McGregor,  1938. 


a(l)  Without  knowledge  of  the  membership  for  past 
years,  (2)  with  correct  knowledge  of  the  membership 
for  the  pest  five  years,  and  (3)  with  Incorrect  knowl¬ 
edge  (figures  five  times  too  large)  of  the  membership 
for  the  past  five  years. 

°The  calculation  of  a  mean  from  these  estimates 
would  hove  been  a  meaningless  operation  because  the 
distribution  revealed  no  central  tendency.  There  were 
"clusters"  of  estimates:  (1)  below  25,000,  (2)  around 
100,000,  (3)  around  500,000,  and  (4)  between  1  and  5 
million. 
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estimates  for  forecasts  and  ratings  are  usually  absent  in  the  Delphi 
tradition,  thus  giving  the  final  results  an  aura  of  precision.  Inter¬ 
quartile  graphs  knock  out  50  percent  of  the  sample  and  the  embarass- 
Ingly  long  tails  of  the  extreme  non-conformists  (e.g.,  see  Fig.  1). 

The  final  report  may  include  a  few  anecdotal  comments  on  selected 
items,  but  rarely  any  connected  discourse  on  controversial  interpreta¬ 
tions.  There  may  also  be  a  few  caveats  on  the  limitations  of  the 
study . 

The  presentation  of  raw  frequency  distributions  of  aggregate  opin¬ 
ion  generates  serious  problems  for  the  user  in  the  interpretation  of 
the  results.  Many  forecasters  may  not  differ  significantly  from  one 
another  with  respect  to  the  null  hypothesis  for  mean  or  median  differ¬ 
ences.  Many  items  may  be  highly  redundant,  with  similar  or  indistin¬ 
guishable  results,  reflecting  a  pervasive  halo  effect.  The  antidote 
is  to  test  far  differences  between  items  in  a  systematic  analysis  of 
variance  for  items,  subjects,  and  rounds,  as  mentioned  earlier,  to 
determine  main  and  interaction  effects.  Redundant  items  can  be  dis¬ 
covered  through  this  technique,  or  through  factor  analysis  of  items, 
as  is  routine  in  conventional  analysis  of  questionnaire  items.  In  an 
unusual  exception  to  standard  Delphi  neglect  of  statistical  analysis, 
Dalkey  and  Rourke  (1971)  used  a  type  of  cluster  analysis  for  quality 
of  life  indicators  which  reduced  a  very  large  number  of  initial  raw 
items  to  a  much  smaller  number  of  relatively  independent  composites  or 
factors.  We  have  no  idea  how  rampant  item  redundancy  and  associated 
halo  effects  are  in  the  results  of  the  Delphi  literature  at  large, 
especially  with  the  characteristic  absence  of  techniques  equivalent  to 
item  factor  analyses.  It  is  easier,  cheaper,  and  perhaps  more  impres¬ 
sive  to  present  the  naive  user  with  unprocessed  raw  data  resting  on 
face  validity. 

After  perfunctory  qualifications,  the  investigator  makes  it  quite 
clear  that  the  experts  have  pronounced  concurred  judgment.  This  is  the 
trump  card  in  the  Delphi  game.  With  the  apparent  tacit  agreement  not 
to  criticise  other  Delphi  investigations,  the  results  tend  to  remain 
unchallenged. 
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4.7.  Delphi  Epistemology 

A  fundamental  epistemological  confusion  exists  between  Delphi 
method  end  Delphi  results.  Practitioners  claim  that  the  end  result  of 
a  Delphi  study  le  a  series  of  expert  forecasts  of  future  events*  or 
more  broadly,  concurred  estimates  of  whatever  social  attribute  is  under 
study.  Prior  discussion  has  provided  grounds  for  a  very  different  in¬ 
terpretation  of  Delphi  results. 

Delphi  items  are  typically  broad,  amorphous  classes  of  events, 
not  precisely  defined  empirical  occurrences.  Delphi  forecasts  are 
opinions  about  such  broad  classes  of  events,  not  systematic,  documented 
predictions  of  such  events.  These  opinions  are  typically  snap  Judg¬ 
ments  frequently  based  on  frea-aasociatlon  stereotypes.  Consensus  for 
such  opinion  tends  to  be  manipulated  consensus  to  minimize  dispersion 
of  opinion.  Further,  the  universe  from  which  items  are  sampled  is 
typically  disregarded  and  unknown,  aa  are  the  identity  and  qualifica¬ 
tions  of  the  expert  panelists. 

Orthodox  Delphi  epistemology  holds  that  the  result  of  this  type 
of  polling  procedure  produces  reasonable  end  useful  forecasts  of  object 
events.  This  worthy  goal  is  not  attained.  The  Delphi  process  produces 
manipulated  convergence  of  opinion  reflecting  ephemeral  attitudes  of 
very  small  samples  of  unknown  individuals.  Here  precisely,  Delphi 
produces  transient  attitudes  about  the  future,  which  is  quite  different 
from  systematic  predictions  of  the  future.  The  epistemological  con¬ 
fusion  arises  from  focusing  on  Delphi  results  and  naively  taking  them 
at  face  value  as  expert  predictions  of  the  future,  rather  than  looking 
at  the  underpinning  method  which  reveals  Delphi  as  an  attitude  polling 
technique  dealing  in  snap  judgments  of  ill-dsfined  issues. 

There  is  a  closely  related  epistemological  issue  concerned  with 
Delphi  validity — the  so-called  accuracy  of  Delphi  predictions.  Ob¬ 
servers  continue  to  say  "How  accurate  is  Delphi?"  "Prove  that  its 
accuracy  is  better  or  worse  then  ot*>er  techniques."  These  questions 
presuppose  s  scientifically  replicable  calender/stopwetch  concept  of 
forecasting  validity  where  an  Impartial  observer  with  e  stopwatch 
welts  for  the  objective  event  to  happen,  clocks  it,  and  records  the 
time  end  date  of  occurrence.  This  is  fine  for  simple,  unambiguous, 
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f Actual  items  such  as  "When  will  man  first  land  and  walk  on  the  moon?" 
The  calendar/stopwatch  concept  can  not  be  applied  to  such  items  as 
"widespread  use  of  robot  services,  automated  rapid  transit,  use  of 
computers  in  tax  collection,  automated  legal  information  retrieval, 
etc."  (see  Fig.  1,  once  more).  There  are  as  many  scenarios  for  each 
of  these  items  as  there  are  respondents.  How  can  anyone  validate  the 
truth  or  falsity  of  an  Inkblot  of  the  future? 

The  way  out  for  some  Delphi  investigators  is  to  ask  the  experts 
at  a  later  date  whether  the  forecasts  have  materialized  (Martino, 

1972).  However,  this  results  in  another  opinion  poll,  or  opinion 
validated  by  opinion,  not  an  objective  assessment  of  external  events. 
This  amounts  to  bootstrap  validation — Delphi  validating  itself.  Such 
studies,  if  conducted  rigorously,  would  provide  an  Indication  of  lon¬ 
gitudinal  test-retest  reliability  (correlation  of  Delphi  with  itself 
over  time),  not  an  indication  of  application  validity  which  requires 
correlation  against  an  external  criterion. 

In  limited  application  areas,  such  as  immediate  or  very  short- 
range  forecasts  (excluding  Che  questionable  applications  to  almanac 
items),  Delphi  accuracy  can  be  measured.  Farquhar’s  study  (1970)  of 
the  estimation  of  software  manpower  requirements,  previously  mentioned, 
is  one  example.  Delphi  performed  very  poorly  when  compared  with  face- 
to-face  groups  in  this  case.  Delphi  forecasting  of  well-defined  short¬ 
term  economic  indices  based  on  Campbell's  (1966)  doctoral  dissertation 
at  UCLA,  was  not  shown  to  differ  substantively  from  simple  extrapola¬ 
tion  of  short-term  time  series  data.  In  1952,  Helmer  published  the 
results  of  a  Delphi  study  predicting  the  results  of  the  1952  presiden¬ 
tial  contest  between  Elsenhower  and  Stevenson.  After  four  rounds, 
the  seven  panelists  converged  on  Stevenson  as  the  winner. 

Evan  thla  very  limited  and  Inconclusive  sample  of  studies  indi¬ 
cates  that  Delphi  results  will  often  be  untrustworthy,  and  will  very 
enormously  between,  and  even  within,  object  problems  or  application 
areas,  raflecting  differences  in  experimenters,  "experts"  selected, 
particularly  with  the  ground  rules  and  baseline  data  made  available 
to  them,  and  numerous  othsr  methodological  issues.  If  these  frequently 
untrustworthy  end  highly  variable  results  over  various  application 
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nreas  are  characteristic  of  relatively  immediate  or  short-range  esti¬ 
mates,  it  can  be  fairly  confidently  inferred  that  stadium  and  longer- 
range  results  will,  by  and  large,  be  even  more  variable  in  reliability 
and  validity.  Recall,  for  example,  Martino's  findings  (1972,  cited 
previously)  that  the  standard  deviations  of  forecasts  for  independent 
studies  are  consistently  highly  and  positively  correlated  with  median 
expected  values  of  occurrence.  (Por  example,  scan  the  widths  of  the 
interquartile  "houses"  in  Fig.  1  with  increasing  estimated  median  of 
occurrence.)  Put  simply,  the  farther  in  the  future  an  event  is  ex¬ 
pected  to  occur,  the  more  uncertain  the  prediction  is  likely  to  be. 

A  concrete  example  illustrates  the  scope  and  magnitude  of  the 
inkblot  problem  for  Delphi  accuracy  or  validity.  Suppose  the  Delphi 
questionnaire  asks  "When  will  mass  information  utilities  become  common¬ 
place?"  The  range  of  "correct"  answers  for  this  item,  depending  upon 
the  scenario  projected  by  the  respondent,  can  literally  vary  from  the 
Western  Renaissance  to  beyond  the  year  2000.  If  "mass  information 
utility"  is  interpreted  to  mean  mass-produced  books,  then  the  answer 
is  sonewhere  in  the  sixteenth  century,  after  the  introduction  and 
spread  of  Gutenberg's  printing  press.  If  interactive  long-distance 
conversation  is  the  preferred  scenario  of  the  respondent,  then  the 
advent  of  the  telephone  In  the  late  nineteenth  century  is  the  answer. 

If  the  expert  interprets  the  item  to  mean  mass  electronic  broadcasting, 
he  would  identify  the  radio  as  the  source,  and  opt  for  the  early 
1920's.  Another  expert  might  interpret  the  item  a3  meaning  audiovisual 
broadcasting,  and  list  the  1950 's  for  the  mass  use  of  television. 

Another  respondent  might  interpret  the  item  as  involving  mass  computer¬ 
ized  transmission  of  information,  and  indicate  the  mid-1970'e  as  the 
point  where  computerized  information  may  greatly  exceed  non-computerlzed 
information  over  various  transmission  media.  If  the  item  were  inter¬ 
preted  as  two-way,  interactive  computer  services  in  the  home,  as  in  * 

Baran's  (1971)  study,  the  respondent  might  pick  the  1980's.  A  cosmo¬ 
politan  expert,  accepting  the  same  scenario,  but  thinking  of  popular 
use  throughout  the  entire  industrialized  world,  would  place  his  pre¬ 
diction  in  the  next  century.  Although  this  illustration  is  deliberately 
extreme,  the  central  point  should  be  quite  clear — the  Delphi 
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questionnaire  format  does  not  lend  Itself  to  scientifically  objective 
and  externally  verifiable  statements  of  future  events. 

In  contrast  to  the  above  example,  consider  more  conventional 
technological  forecasting  studies  under  the  sponsorship  of  NASA 
(Feldman,  1965)  and  the  Air  Force  in  Project  Forecast  (Aasler  and 
Newton,  1963).  In  both  of  these  studies,  the  authors  assembled  exten¬ 
sive  data  on  engineering  characteristics  for  specialized  forecasting 
targets — communication  satellite  output  devices  (Feldman) ,  and  multi¬ 
purpose  long  endurance  aircraft  (Aasler  and  Newton).  Qualifying  speci¬ 
fications  and  assumption?  were  spelled  out,  technical  baselines  were 
carefully  defined  and  established,  and  most  likely  technological  de¬ 
velopments  were  projected.  Most  results  were  expressed  quantitatively, 
often  in  graphic  format.  The  key  difference  between  these  results  and 
conventional  Delphi  results  lies  in  the  rigorous  technical  framework 
in  which  the  forecasts  were  embedded.  These  NASA  and  Air  Force  ex¬ 
amples  illustrate  initial  steps  in  the  direction  of  operationally  de¬ 
fined  predictions  essential  for  scientifically  verifiable  forecasts. 

Thus,  when  someone  asks  "How  accurate  are  Delphi  results?"  the 
answer  should  be  that  "Accuracy  can  not  be  measured  for  most  Delphi 
items,  because  changing  attitudes  and  opinions  on  amorphous  issues 
are  not  true  or  false  and  do  not  have  specific  dates  at  which  they 
occur."  Asking  for  proof  or  disproof  of  Delphi  accuracy  amounts  to 
giving  Delphi  '•redit  for  generating  results  capable  of  proof — a  prop¬ 
erty  that  conventional  Delphi,  as  currently  practiced,  does  not 
possess . 

There  is  nothing  inherently  wrong  with  studying  and  learning  more 
about  opinions  concerning  the  future.  Such  knowledge  is  crucial  to 
any  intelligent  appraisal  of  the  future.  Rut  we  should  not  confuse 
such  opinion  with  seriously  considered,  qualified  and  documented  pre¬ 
dictions  of  well-defined  future  developments.  Attitudes  and  opinions 
change  and  fresh  sampling  in  real  time  is  needed  to  track  such  changes. 
And  the  smopllng  must  be  explicit  in  terms  of  subject  populations  if 
any  systematic  Inferences  are  to  be  made. 
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A.8.  Anonymity  and  Accountability 

The  anonymity  of  Delphi  experts  serves  the  duel  purpose  of  attract¬ 
ing  expert  panelists  by  guaranteeing  protection  against  lndlvlduel  ac- 
countabiU ty,  and  projecting  an  inviting  image  of  a  kind  of  permissive 
brainstorming  where  "anything  goes"  to  help  "cream  off"  the  best  the 
experts  have  to  offer.  The  panelists  are  assured  full  protection 
against  any  invasion  of  privacy.  When  coupled  with  the  blandishments 
of  joining  the  inner  circle  of  eminent  experts,  the  combination  is 
hard  to  resist.  But  few  have  realised  that  the  price  of  such  induce¬ 
ments  is  abandonment  of  accountability,  and  may  promote  elltlat  vested 
interests . 

Under  a  "no  disclosure  of  nmses"  policy,  no  individual  is  account¬ 
able  for  either  his  own  responses  or  for  group  Delphi  results.  As 
pointed  out  eerlier,  Delphi  eabodies  circular  buck-passing.  The 
director  reports  group  opinion  following  an  objective  ritual;  the  re¬ 
sults  ere  cot  his  personal  opinion.  Each  panelist  is  faceless  in  any 
of  the  results,  end  can  always  blame  nameless  others  for  any  findings 
he  dislikes.  The  consumer  of  Delphi  gets  his  low-cost  preview  of  the 
future,  and  can  claim  ha  had  nothing  to  do  with  the  final  results. 

Directors  should  be  accountable  for  all  flawe  in  the  method,  and 
for  implicitly  or  explicitly  overstating  the  value  and  significance  of 
potentially  misleading  final  results.  Panelists  should  be  accountable 
for  unwittingly  lending  the  authority  of  their  reputations  end  their 
support  to  demonstrably  unreliable  and  invalid  short-cuts  to  the  future. 
Individual  and  institutional  users  should  be  accountable  for  funding 
and  popularising  such  studies,  and  for  accapting  Delphi  forecasts  at 
face  value. 

Elltlat  tendencies  are  strongly  reinforced  not  because  of  any 
diabolical  plot  on  the  part  of  Delphi  investigators,  but  for  the  more 
mundane  and  more  compelling  reason  that  it  is  a  lot  sealer  and  faatar 
to  asse^le  colleagues,  acquaintances,  or  second-order  recommended 
acquaintances  for  the  expert  panel. 

A  major  attraction  of  Delphi  for  busy  researchers  of  all  callings 
is  that  it  is  cheap  and  easy,  as  wall  as  a  relatively  painless  and 
well-protected  technique.  A  study  can  be  conducted  end  e  paper  produced 
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with  relatively  small  effort.  Martino  (1972)  claims  that  M.  .  .a 
planning  factor  of  two  professional  manhours  per  panelist  per  question¬ 
naire  is  a  fair  approximation  to  the  workload  which  will  be  required" 

(p.  60). 

The  questionnaire  is  quite  likely  to  represent  many  aspects  of  the 
^fork  done  by  these  experts,  almost  by  definition.  Chances  are  that 
while  such  panelists  will  have  much  to  disagree  over,  most  will  be  in¬ 
terested  in  promoting  the  image,  value,  and  particularly  the  future  of 
their  field.  Big  developments  will  then  be  perceived  as  occurring 
early  and  making  large  impacts  on  society.  For  example,  Nanus,  Wooten 
and  Borko  (1973),  in  their  Delphi  study  on  the  social  implications  of 
multinational  computer  systems,  admit  that  their  sample  of  56  "eminent" 
panelists,  typically  active  in  various  aspects  of  this  field,  were 
probably  biased  to  some  extent  toward  promoting  the  importance  and  en¬ 
hancing  the  image  of  multinational  computer  systems.  There  is  no  ma¬ 
levolent  design  or  covert  collusion  in  such  opinions,  merely  self- 
aggrandizement  and  self-interest. 

Controlled  experiments  soliciting  opinions  from  contrasting  or 
even  antagonistic  groups  (expert  or  otherwise)  are  likely  to  produce 
quite  different  results.  As  mentioned  earlier,  Delphi  opinion  polls 
measure  attitudes  toward  future  events,  not  predictions  of  such  events 
In  their  awn  right.  Ab  currently  practiced,  Delphi  can  easily  slant 
results  in  the  direction  of  aggrandizing  vested  Interests.  With  anon¬ 
ymous  sampling  of  "experts,"  the  burden  of  proof  should  be  on  the 
Delphi  investigator  to  demonstrate  that  his  panel  does  not  represent  a 
narrow  elitist  circle. 

Kopklnd  (1967)  ,  in  his  widely  cited  article  ou  "Ihe  Future  Plan¬ 
ners, "  expressed  his  concern  over  futurist  elitism.  "The  danger  is 
that  Government  and  corporate  elites  will  monopolize  the  business  of 
question-asking,  and  so  manipulate  the  attitudes  of  society  they  are 
pretending  to  serve  as  disinterested  technicians"  (p.  23). 

A. 9.  Adversary  Process 


Moat  Delphi  practitioners  claim  that  Delphi  is  able  to  go  where 
other  investigators  fear  to  tread.  Opinion  can  peer  into  every  nook 
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and  cranny,  particularly  those  inaccessible  to  conventional  techniques, 
Delphi  thus  has  the  advantage  of  being  able  to  get  "there"  first,  or 
among  the  first,  and  of  asking  early  pronouncements  concerning  new 
horizons  far  in  the  future.  This  capitalization  on  novelty  is  part  of 
the  dramatic  appeal  of  Delphi.  Pluablng  the  depths  and  climbing  the 
heights  of  the  future  hold  spills,  thrills,  chills,  and  some  jolts  of 
future  shock  for  everyone. 

It  would  seem  plausible  that  at  least  until  we  learn  a  good  deal 
about  any  new  domain,  it  should  be  the  object  of  free  Inquiry  and  of 
very  active  adversary  proceedings.  Delphi  systematically  inhibits  the 
adversary  process.  This  indictment  is  not  in  any  sense  original  with 
the  author.  As  cited  in  various  contexts  throughout  this  critique, 
variations  of  this  indictment  have  been  made  by,  Bedford  (1972), 
Milkovich,  Ann on 1  and  Mahoney  (1972),  Turoff  (1972),  and  Weaver  (1970). 

Delphi  deliberately  factors  out  face-to-face  confrontation,  and 
the  adversary  process  associated  with  it,  as  one  of  its  prime  philos¬ 
ophical  tenets  Justifying  efficient  consensus.  Arguments  are  filtered, 
buffered,  and  effectively  neutralized  in  Delphi.  A  panelist  can  par¬ 
ticipate  without  providing  any  justification  for  any  of  his  opinions 
throughout  the  entire  procedure.  More  conscientious  panelists  provide 
occasional  brief  commentaries. 

The  real  payoff  for  the  Delphi  investigator  is  obtaining  maximum 
consensus  from  the  experts.  Interquartile  Delphi  forecasting  graphs, 
spreading  from  now  to  never,  are  the  nemesis  of  Delphi  practitioners. 
The  smaller  the  spread  the  more  powerful  the  impact.  Real  adversary 
excitement  over  authentic  controversial  issues  is  plainly  the  enemy  of 
consensus.  Boredom  and  snap  responses  make  for  smaller  differences 
and  maximum  consensus.  In  many  cases,  only  the  outliers  have  to  jus¬ 
tify  their  positions  in  Delphi  iteration;  directors  make  minimal  de¬ 
mands  on  those  occupying  the  middle  ground. 

By  inhibiting  the  adversary  process,  Delphi  also  inhibits  open 
exploration  of  new  domains.  Free  exploration  leads  to  adversary  in¬ 
quiry  and  generates  new  controversy.  This  can  lead  to  polarization  of 
opinion  that  undermines  consensus  in  final  Delphi  results.  But  it  is 
priolsely  the  new  domains  that  need  free  exploration  and  the  adversary 
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process  the  Most.  Delphi  should  be  prodding  conformers  and  rewarding 
outliers  to  maximize  exploration,  highlight  controversy,  and  nap  out 
the  unknown.  When  we  are  really  Ignorant  we  need  all  the  contrasting 
viewpoints  we  can  get  to  encourage  free  and  informed  choice. 


4.10.  Delphi  Isolationism 

The  history  of  Delphi  reveals  a  highly  exploratory  and  tentative 
technique  that  was  never  validated.  Delphi  was  obviously  full  of  prob¬ 
lematic  issues  and  potentially  serious  flaws,  and  was  treated  with  some 
measure  of  caution  and  skepticism  by  its  Rand  originators  before  the 
Gordon-Helmer  study  (1964)  catapulted  the  technique  into  international 
prominence.  After  that  point,  the  shaky  hypotheses  on  which  Delphi 
rested  were  apparently  transformed  into  axioms,  and  Delphi  wan  promoted 
as  an  established,  proven  technique. 

Only  relatively  recently  have  Dalkey  and  same  of  hio  co-vorkers 
made  attempts  to  demonstrate  the  validity  of  Delphi,  as  reviewed  in 
this  report,  primarily  with  almanac-type  items  and  non-expert  panelists 
such  as  college  students.  These  efforts,  and  spotty  returns  from  a 
small  number  of  other  studies  mentioned  in  this  review,  provide  no 
scientific  validation  of  Delphi.  This  history  of  early  experimentation 
and  tardy  efforts  to  assess  validity  reflects  a  pattern  of  isolationism 
from  the  mainstream  of  behavioral  research. 

Delphi  has  led  a  protected  existence  for  the  decade  it  has  been 
actively  pursued.  Prom  exploratory  and  tentative  beginnings  at  Rand, 

It  has  spread  from  government  to  industry  and  academia,  and  diversified 
from  scientific  and  technological  forecasting  to  policy  studies  and 
planning,  to  quality  of  life  assessment,  and  is  being  touted  as  the 
emerging  nexus  for  human  communication  and  decision  making  (e.g., 
Turoff,  1972).  Droves  of  eminent  people  and  experts  from  all  callings 
have  lent  their  nme,  time,  and  effort  to  hundreds  of  Delphi  investiga¬ 
tions.  All  this,  and  undoubtedly  more  to  come.  Why? 

In  part,  because  there  has  been  virtually  no  critical  literature. 
The  roots  of  this  criticism-free  development  of  Delphi  are  found  In 
two  sources — the  Isolation  of  Delphi  from  the  mainstream  of  relevant 
behavioral  science,  end  the  rapid  concurrent  emergence  and  growth  of 
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futurlem.  For  various  reasons,  Delphi  originators  and  subsequent  Delphi 
practitioners  have  shied  away  from  psychometric  and  opinion  survey 
specialists  who  could  have  professionalised  Delphi  as  an  opinion  polling 
technique  along  the  lines  previously  suggested  in  connection  with  the 
discussion  of  the  APA  test  annual  and  social  science  standards.  The 
existence  of  this  isolation  is  attested  by  the  fact  that  there  are 
virtually  no  listings  of  Delphi  studies  in  the  Psychological  Abe  tracts 
as  revealed  by  our  literature  review.  The  proof  of  this  isolation  is 
the  disregard  and  unconcern  for  professional  questionnaire  standards 
in  Delphi  practice  that  has  been  heavily  documented  in  this  study. 

The  reasons  for  such  isolation  are  not  hard  to  find.  The  profes¬ 
sional  standards  would  iaaediately  transform  Delphi  from  a  cheap  and 
easy,  short-cut  technique  to  a  far  sore  difficult,  expansive  and  time- 
consuming  procedure.  Unprepared  and  untrained  Delphi  Investigators 
would  have  to  develop  new  skills  in  psychometrics ,  opinion  sampling 
and  polling,  and  experiaental  design  with  human  subjects,  and  would 
lose  considerable  control  over  the  technique  if  experts  In  these  skill 
aress  were  taken  seriously. 

Delphi  practitioners  and  many  futurists,  broadly  considered,  iden¬ 
tify  themselves  as  interdlsclpllnarlana .  They  sought  to  enlist  the 
necessary  diversity  of  skills  to  asseasaents  of  the  future.  This  is 
most  commendable  if  taken  seriously.  The  place  to  begin,  however,  is 
with  the  disciplines  vital  to  the  method.  This  was  never  done  with 
Delphi. 

Neither  the  originators  of  Delphi,  nor  subsequent  practitioners, 
have  been  willing  to  attempt  to  establish  rigorous  standards,  and  to 
police  the  Delphi  literature  by  discriminating  between  better  and 
poorer  work.  This  has  contributed  to  the  spate  of  crude  Delphi  studies 
generated  by  neophytes. 

This  leek  of  standards  Is  characteristic  of  new  disciplines  going 
through  early  growth.  Futurism  has  not  been  heavily  pursued  for  much 
more  than  e  decade.  Delphi  played  no  small  part  in  getting  futurists 
on  the  map  by  dignifying  forecasting  with  its  seemingly  impressive 
ritual  for  obtaining  expart  consensus .  Other  methods,  such  as  brain¬ 
storming,  scenarios,  gmaing,  Input-output  analyses ,  contextual  mapping, 
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slmulat.lan,  and  Morphological  analyses  also  experienced  rapid  growth 
during  this  period,  contributing  largely  undisciplined  exploratory 
techniques  to  futures  forecasting  and  planning.  Each  technique  needs 
adversary  checks  and  balances  for  healthy  growth,  and  futurism  as  a 
whole  needs  to  develop  minimal  professional  standards  and  a  vigorous 
critical  literature  representing  aore  authentic  Interdisciplinary  work. 


4.11.  Results  of  the  Analysis 

This  portion  of  the  study,  concerned  with  analysis  of  the  specific 
and  unique  assumptions  and  principles  of  Delphi,  as  distinct  from  opin¬ 
ion  questionnaires  and  human  experimentation  broadly  considered,  was 
organised  under  10  key  questions  formulated  at  the  outset.  The  anal¬ 
ysis  suggests  the  following  answers  to  the  10  questions  for  conventional 
Delphi: 


1.  The  Delphi  concept  of  the  expert,  and  Its  claim  to  represent 
valid  expert  opinion,  is  scientifically  untenable  and  over¬ 
stated.  As  summarized  by  Professor  Hay thorn,  an  external 
technical  reviewer  of  this  report,  "...  the  procedure  by 
which  the  selection  of  subjects  occurs  Is  not  properly  expli¬ 
cated,  the  exact  nature  of  the  panel  of  experts  is  often  left 
unspecified,  and  the  implicit  assumption  that  results  obtained 
using  conventional  Delphi  with  a  panel  of  experts  is  better 
than  or  different  from  results  that  would  be  obtained  using 
another  population  has  not  been  empirically  established." 

2.  Delphi  claims  of  the  superiority  of  group  over  individual 
opinion,  and  of  the  superiority  of  remote  end  private  opinion 
over  face-to-face  encounter,  as  well  as  their  counter- 
stetemants,  ere  unproven  generalizations. 

3.  Delphi  consensus  is  spacious  consensus.  As  succinctly  stated 
by  Professor  Hey thorn,  ".  .  .  the  group  process  used  in  Delphi 
rounds  is  quits  similar  to  the  techniques  used  In  social 
psychological  research  to  study  group  conformity,  rejection 

of  deviant  opinion,  and  dalndlvlduallzatloo ,  all  of  which  have 
basn  shown  to  be  counterproductive  with  regard  to  the  quality 
of  gToup  decisions." 
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4.  Delphi  questions  are  likely  to  be  vague. 

5.  Delphi  responses  are  likely  to  be  ambiguous. 

6.  Delphi  results  probably  represent  compounded  ambiguity. 

7.  Delphi  is  primarily  concerned  with  transient  collections  of 
snap  judgment  opinions  of  polled  individuals  from  unknown 
samples,  which  should  not  be  confuBed  or  equated  with  co¬ 
herent  predictions,  analyses,  or  forecasts  of  operationally 
defined  and  systematically  studied  behaviors  or  events. 

8.  Delphi  anonymity  reinforces  unaccountability  in  method  and 
findings . 

9.  Delphi  systematically  discourages  adversary  process  and  in¬ 
hibits  exploratory  thinking. 

10.  Delphi  has  been  characterized  by  isolation  from  the  main¬ 
stream  of  scientific  questionnaire  development  and  behavioral 
experimentation,  and  has  set  an  undesirable  precedent  for 
interdisciplinary  science  in  the  professional  planning  and 
policy  studies  community. 
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5.  EPILOGUE 


The  following  16  conclusions  sum  up  the  evaluation  of  conventional 
Delphi  In  ragard  to  Its  nethod  and  application.  This  report  finds  con¬ 
ventional  Delphi : 

1.  Often  characterized  by  crude  questionnaire  design. 

2.  Lacking  in  minimal  professional  standards  for  opinion  Item 
analyses  and  pilot  testing. 

3.  Highly  vulnerable  on  its  concept  of  "expert"  with  unaccount¬ 
able  sampling,  and  in  the  selection  of  panelists,  expert  or 
otherwise. 

4.  Abdicating  responsibility  for  item  population  sampling  in 
relation  to  theoretical  constructs  for  the  object  area  of 
inquiry. 

5.  Virtually  oblivious  to  reliability  measurement  and  scientific 
validation  of  findings. 

6.  Capitalizing  on  the  fallacy  of  the  expert  halo  effect. 

7.  Typically  generating  snap  answers  to  ambiguous  questions  re¬ 
presenting  inkblots  of  the  future. 

6.  Seriously  confusing  aggregations  of  raw  opinion  with  system¬ 
atic  prediction. 

9.  Capitalizing  on  forced  consensus  based  on  group  suggestion. 

10.  Unwittingly  inhibiting  individuality  and  any  adversary  process 
by  overtly  and  covertly  encouraging  conformity  and  penalizing 
the  dissident. 

11.  Reinforcing  and  institutionalizing  premature  closure,  using 
a  highly  questionable  ritual  for  conducting  opinion  studies 
that  tends  to  inhibit  more  scientific  approaches. 

12.  Giving  an  exaggerated  Illusion  of  precision,  misleading  un¬ 
informed  users  of  results. 

13.  Indifferent  to  and  unaware  of  related  techniques  and  findings 
in  behavioral  science  in  such  areas  as  projective  techniques, 
psychometrics,  group  problem  solving,  and  experimental  design. 
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14.  Producing  virtually  no  serious  critical  literature  to  teat 
basic  assumptions  and  alternative  hypotheses. 

15.  Denigrating  group  and  face-to-face  discussion,  and  claiming 
superiority  of  anonymous  group  opinion  over  competing  ap¬ 
proaches  without  supporting  proof. 

16.  Encouraging  a  short-cut  social  science  method  that  is  lccklng 
in  minimum  standards  of  professional  accountability. 

5.1.  Final  Evaluative  Recommendations 

Two  alternative  final  recommendations  were  considered  as  conclu¬ 
sions  of  this  evaluation.  One  wan  to  seek  to  upgrade  Delphi  by  recom¬ 
mending  higher  standards,  more  consistent  with  scientific  method  in  the 
collection,  analysis,  and  use  of  questiounaire  data.  The  other  was  to 
conclude  that  the  assumptions  and  principles  on  which  conventional 
Delphi  is  based  are  so  unscientific  and  inherently  ruisleading  that  they 
preclude  any  attempts  to  improve  the  technique.  This  second  alterna¬ 
tive  was  tantamount  to  a  recommendation  to  drop  Delphi  completely. 

The  evidence  adduced  in  this  study  clearly  indicates  that  the 
massive  liabilities  of  Delphi,  in  principle  end  in  practice,  outweigh 
its  highly  doubtful  assets. 

As  the  preferred  alternative  to  conventional  Delphi,  professionals, 
funding  agencies,  and  users  are  urged  to  work  with  psychcmetrically 
trained  social  scientists  who  can  apply  rigorous  questionnaire  tech¬ 
niques  and  scientific  human  experimentation  procedures  tailored  to 
their  particular  needs.  It  is  recommended  that  conventional  Delphi  be 
dropped  from  institutional,  corporate,  and  government  use  until  its 
pri  nciples,  methods,  and  fundamental  applications  can  be  experimentally 
established  as  scientifically  tenable. 

5.2.  Beyond  Delphi 

Some  will  grant  the  very  shaky  oplnionative  structure  of  Delphi, 
and  Insist  that  Delphi  was  never  really  put  forth  as  science,  but  merely 
as  a  heuristic  vehicle  for  exploring  vague  and  unknown  future  issues 
otherwise  Inaccessible.  They  might  insist  that  Delphi  as  an  exercise 
has  generated  many  insights  and  has  been  well  received. 
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Jolaon  and  Rossow  (1971)  have  commented  on  the  heuristic  value  of 
Delphi  In  facilitating  communication  Li  the  corporate  environment. 

Relaaan  et  al.  (1969)  have  noted  the  coaaunicatlon  potentials  of  Delphi 
for  community  participation  in  evaluating  alternative  social  services. 
Even  as  a  heuristic  exercise,  it  would  be  highly  advisable  to  mix 
Iterative  polling  with  varying  forms  of  quantitative  and  qualitative 
feedback,  personal  confrontation  where  feasible,  cultivated  development 
of  adversary  positions  as  opposed  to  consensus,  and  controlled  varia¬ 
tions  In  the  type  and  level  of  anonymity.  As  we  have  seen,  there  is 
nothing  sacred  in  the  Delphi  process — all  basic  assianptions ,  particu¬ 
larly  In  informal  exercises,  should  be  systematically  challenged,  ex¬ 
amined,  and  tested  with  other  eclectic  approaches,  and  tailored  to  the 
unique  mission  and  needs  of  the  object  problem. 

Brainstorming,  if  done  properly,  is  fun,  generates  many  insights, 
and  can  be  well  received.  Advocates  of  brainstorming  no  longer  present 
their  results  as  finished  products.  Practitioners  of  Delphi  publish 
results  in  journals,  as  master’s  and  doctoral  dissertations  (e.g., 
Kochman,  1968,  Campbell,  1966,  and  Weaver,  1969)  as  major  corporate 
reports  (e.g.,  North  and  Pyke,  1968),  as  significant  social  Indicators 
for  national  and  international  planning  (e.g.,  Dalkey,  Rourke,  Lewis 
and  Snyder,  1972,  BJerrum,  1968),  as  results  worthy  of  weighty  con¬ 
sideration — the  embodiment  of  balanced  expert  opinion. 

There  is  a  vast  difference  between  Delphi  as  an  informal  forecast¬ 
ing  exercise  among  questionnaire  respondents,  and  Delphi  as  the  authen- 
tlc  embodiment  of  thoughtfully  concurred  expert  opinion  wherever  It  Is 
applied.  Nanus,  Wooten,  and  Borko  (1973)  in  a  relatively  ambitious 
Delphi  study  on  ths  social  impact  of  the  multinational  computer,  make 
it  clear  that  no  clalma  are  made  for  the  reliability  or  validity  of 
their  Delphi  reaulta — Delphi  was  used  for  strictly  exploratory  purposes 
In  an  uncharted  domain,  ".  .  .  the  authors  chose  to  use  Delphi  with 
realisation  that  the  results  would  be  more  in  the  nature  of  a  structured 
’b rains toralng’  session  with  noted  thinkers  than  a  scientific  exercise 
in  prediction"  (p.  11). 

The  rejection  of  conventional  Delphi  recommended  here  should  not 
in  any  way  be  construed  as  denying  the  growing  end  urgent  need  of  society 
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to  learn  and  understand  more  about  the  future.  Perhaps  the  greatest 
of  all  human  rights  is  the  right  to  help  shape  and  determine  one's  own 
and  society's  future.  We  need  to  know  far  more  about  hicnan  attitudes 
toward  future  developments . 

It  is  to  be  hoped  thet  forthcoming  opinion  polls  will  systemati¬ 
cally  sample  attitudes  toward  the  future  from  all  segments  of  the  popu¬ 
lation  for  more  effective  end  more  humanietlcally  informed  social  plan¬ 
ning.  Delphi,  with  its  exclusive  reliance  on  small  coteries  of  "ex¬ 
perts,"  hes  unwittingly  fostered  another  fora  of  elitism  to  set  the 
pace  and  formulate  the  pattern  for  attitudes  toward  the  future. 

The  originators  of  Delphi  had  the  right  instincts  in  responding 
to  growing  and  pressing  needs  to  enlist  the  active  participation  of 
geographically  distributed  professionals  to  work  In  concert  assessing 
unknown  and  complex  problems.  Perhaps  their  most  significant  Insight 
was  the  concept  of  physically  distributed  teams  building  a  emulative 
base  of  knowledge  through  the  mechanism  of  temporally  spaced  interac¬ 
tion  and  feedback.  Although  this  concept  responds  to  a  strongly  felt 
social  need,  the  implementation  has  been  counterproductive.  The  orig¬ 
inators  arrived  at  premature  closure  along  the  lines  of  an  iterative 
ritual  producing  ambiguous  results. 

Instead  cf  testing  a  great  variety  of  flexible  alternatives,  the 
method  zeroed  in  on  iterative  statistical  group  response.  The  alter¬ 
natives  could  have  branched  out  into  structured  adverser/  procedures 
including  dialectical  planning  (e.g.,  Mason,  1969),  adversary  polling 
between  groups  with  vested  Interests  as  in  SPRITE  (e.g.,  lledford,  1972), 
iterative  online  teleconferences  (e.g.,  Sackman  and  Cltrenbam,  1972), 
and  eclectic  mixtures  of  confrontation  and  isolated  responnes  (e.g., 
Heller,  1969,  Weaver,  1972).  All  of  these  areas  need  vigorous  experi¬ 
mental  work. 

It  is  beyond  the  scope  of  this  analysis  to  enter  into  n  systematic 
review  of  areas  of  inquiry  related  to  Delphi  and  possible  offshoots 
that  might  lead  to  useful  advances  in  method  and  findings.  Suffice  it 
to  say  that  many  research  opportunities  exist  for  teleconferencing, 
iterative  polling,  the  analysis  of  human  attitudes  toward  the  future, 
cooperative  problem  solving  among  geographically  dispersed  individuals, 
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«nd  the  social  dynamics  of  real,  not  specious  consensus  which  should 
be  based  on  a  profound  understanding  of  the  adversary  process  in  its 
own  right. 

Consumers  of  information  on  the  future  need  far  better  advice  and 
protection  from  contributing  professionals  than  they  have  gotten  to 
date.  The  future  is  far  tuo  important  for  the  human  species  to  be  left 
to  fortune  tellers  using  new  versions  of  old  crystal  balls.  It  is  time 
for  the  oracles  to  move  out  and  for  science  to  novo  in. 


Practiiiig  pm  Mink 
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Appendix 

S SMI -ANNOTATED  DELPHI  BIBLIOGRAPHY 


This  bibliography  Includes  standard  and  annotated  citations  to  the 
Delphi  literature.  Much  of  this  material  was  assembled  by  Barbara 
Quint  of  the  Rand  Library  staff.  Annotations  are  included  as  available 
from  our  sources;  entries  are  arranged  alphabetically  by  author. 

We  were  greatly  aided  by  Delphi  listings  made  available  to  us  from 
the  following  sources : 

1.  Delphi  and  Long-Range  Forecasting ,  SB-1019,  The  Rand  Corpora¬ 
tion,  Santa  Monica,  California,  1972.  (Annotations  frou  this 
source  are  indicated  by  (R)  at  Lhe  end  of  the  listing.) 

2.  Selected  Bibliography  on  Delphi  Literature ,  Institute  for  the 
Future,  Menlo  Park,  California,  1972. 

3.  Pill,  Juri,  "The  Delphi  method:  Substance,  context,  a  cri¬ 
tique  and  an  annotated  bibliography,"  Socioeconomic  Planning 
Science,  Vol.  5,  pp.  57-71,  1971.  (Annotations  from  this 
source  are  indicated  by  (P)  at  the  end  of  the  listing.) 

4.  Turoff,  Murray,  "Delphi  and  its  potential  impact  on  informa¬ 
tion  systems,"  AFIPS  Conference  Proceedings ,  Vol.  39,  AF1PS 
Press,  Montvale,  New  Jersey,  pp.  317-326,  1971. 

5.  Annotated  Delphi  Bibliography  ,  provided  by  Michael  T.  Bedford, 
Bell  Canada  Business  Planning  Croup,  Montreal,  Canada,  1973. 
(Annotations  from  this  source  are  indicated  by  (B)  at  the  end 
of  the  listing.) 

6.  A  search  through  various  standard  indexes  In  the  Rend  library. 

The  Bell  Canada  bibliography  and  the  Rand  bibliography  provided 
the  most  extensive  annotated  listings.  The  primary  focus  of  the  Bell 
Canada  entries  is  on  corporate  applications  of  Delphi.  These  entries 
Include  listings  and  cross-references  for  corporations  using  Delphi 
which  are  retained  for  the  convenience  of  the  reader.  The  Rand  en¬ 
tries  prlmailly  cover  the  historical  and  methodological  literature. 
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ADELS  ON,  M. ,  M.  ALKIN,  C.  CASEY ,  And  0.  HELMER 

"Planning  Education  for  the  Puture:  Coenents  on  a  Pilot  Study" 

American  Behavioral  Scientist ,  Vol.  10,  No.  7,  1967. 

The  character  of  Aaericon  education  la  determined  by  many  related 
decisions,  improving  it  will  require  a  broad  base  of  participation 
within  and  outside  of  school  systems.  This  requirement  implies  a  need 
for  generating  and  disseminating  information  about  education,  and  for 
devising  procedures  for  bringing  informed  judgment  to  bear  on  the  de¬ 
cision  process  in  a  regularized  way.  It  may  be  as  important  to  improve 
the  decision  process  in  education  as  to  modify  any  of  the  specific  fea¬ 
tures  of  contemporary  schooling.  The  trend  toward  systematizing  or  ra¬ 
tionalizing  the  decision  process  seems  promising,  although  there  is  a 
need  to  avoid  centralized  control  of  the  process  of  developing  new  cit¬ 
izens  who  are  to  live  in  a  democratic  society.  The  future  role  of  the 
federal  government  in  American  education  is  one  of  the  deep  residual 
issues . 

AIL 

See  paper  by  PACKARD  that  describes  the  use  of  Delphi  at  AIL  for 
forecasting  the  development  in  the  LSI  chip  industry, 

ALDERSON,  R.  C. ,  and  W.  C.  SPROULL 

"Requirement  Analysis,  Need  Forecasting,  and  Technology  Planning  Using 
the  Honeywell  PATTERN  Technique" 

'Technological  Forecasting  and  Social  Change,  Vol  3,  No.  3,  1972,  pp. 
255-265. 

The  authors  describe  the  development  and  use  of  the  Honeywell  PAT¬ 
TERN  technlque--Planning  Assistance  Through  Technical  Evaluation  of 
Relevance  Numbers.  This  technique  may  be  regarded  as  a  distant  cousin 
to  Delphi  alnce  groups  of  experts  are  used  to  develop  consensus  on  the 
relevance  nuabere  for  the  projects  under  consideration.  Although  the 
technique  was  developed  for  military  purposes,  the  article  uses  examples 
of  a  personal  transportation  decision  and  a  bio-medical  study  conducted 
by  Honeywell.  (B) 

ALLPORT,  GORDON 

Becoming 

Yale  l' diversity  Press,  New  Haven,  1955. 

AMABA,  R.  C.,  A.  J.  LIP1NSKI 

"Some  Views  on  the  Use  of  Expert  Judgment” 

Technological  Forecasting  and  Social  Change,  Vol.  3,  No.  3,  1972. 

AMENT,  R.  H. 

"Comparison  of  Delphi  Forecasting  Studies  in  1964  and  1969" 

Futures,  Vol.  2,  Mo.  1,  March  1970.  (Also  Institute  for  the  Future, 
P-9.) 
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AME RICAN  PSYCHOLOGICAL  ASSOCIATION 

Standards  for  Educational  and  Psychological  Teats  and  Manuals 

Washington,  0.  C.,  1966. 

AMSLER,  R.  C.,  and  J.  S.  NEWTON 

Multipurpose  Long  Endurance  Aircraft  (MPLE)  Airplane  Design  Analysis 

Northrop  Corporation,  NOR-6 3-109 ,  June  1963. 

ANAS T  AS 1 ,  A. 

Psychological  Testing 

Third  Edition,  Macmillan,  New  York,  1968. 


A.T.6T. 

The  Future  of  the  Telephone  Industry 

Sponsor  of  the  Institute  for  the  Future  Study  R-20,  (see  BARAN  and 
LIPlNS.VY) . 

A.T.&T. ,  and  WESTERN  ELECTRIC 

Communications  Needs  of  the  Seventies  and  Eighties 

Deceaber  1971.  (Internal  document.) 

Reports  the  results  of  a  Delphi  study  sponsored  by  the  above  com¬ 
panies  .  High  priority  communications  needs  sre  described  In  13  cate¬ 
gories.  These  needs  are  ranked  by  categories.  (B) 

AYRES,  ROBERT  UL 

Technological  Forecasting  and  Long-Range  Planning 

McGraw-Hill  Book  Coapany,  New  York,  1969. 

BARAN,  P. 

Potential  Market  Demand  for  Tuo-Uay  Information  Service  to  the  Home 

Institute  for  the  Future,  R-26,  1971. 

BARAN,  P. 

The  Future  of  Neosprint 

Institute  for  the  Future,  September  1971. 

McMillan  Bloedel  Ltd.  eponsored  this  study  which  was  designed  to 
develop  a  better  underetending  of  the  factors  that  will  have  a  signifi¬ 
cant  Impact  on  the  demand  for  newsprint  and  newspapers  in  the  next 
thirty  years.  The  study  examines  the  future  of  newsprint  and  substitute 
media:  non-wood  fiber  paper,  magazine*  and  books,  electronic  eysteaa 
(TV  and  CATV).  After  an  examination  of  the  newspaper  business,  the 
study  exaalnes  some  broader  issues:  ecological  cons Ida rat lone,  Inter¬ 
national  coaaerce  and  the  U.S.  economy.  Many  charts  of  the  specific 
results  are  Included.  (B) 

BARAN,  P.,  and  A.  J.  LIPINSK1 

The  tut  are  uj'  the  Telephone  Industry 

Institute  for  the  Future,  K-20,  Sept saber  1971. 

This  A.T.6T.  sponsored  study  examines  five  mein  areas  that  will 
have  significant  Impact  on  the  future  of  the  communications  business. 
These  are:  regulation,  social  change,  existing  services  end  networks, 
new  services  and  networks,  labor  force  and  urban  change.  The  five 
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panels  consisted  of  210  respondents  from  Inside  and  outside  the  Bell 
System.  (B) 

BEDFORD,  MICHAEL  T. 

A  Technology  Assessment  of  Future  Home  Communications  Services,  A  Study 
Pix>po8al 

Bell  Canada,  Business  Planning  Paper  No.  12,  May,  1973. 

BEDFORD,  Michael  T. 

"The  Value  of  'Comments'  Analysis  and  an  Analysis  of  SPRITE  as  a  Plan¬ 
ning  Tool" 

Delphi:  The  Bell  Canada  Experience,  Bell  Canada,  October  1972. 

This  article  examines  the  use  of  the  Delphi  technique  In  the  cor¬ 
porate  environment.  The  paper  indicates  that  a  broad  range  of  Delphi 
studies  have  been  conducted  or  sponsored  by  various  corporations  in 
North  America  and  Europe.  These  studies  were  conducted  by  consultants 
or  planning  groups  in  the  business  fires.  Four  Delphi  planning  groups 
conducted  by  Bell  Canada's  Business  Planning  Group  are  examined.  Re¬ 
sults  from  these  studies  are  illustrated.  Several  important  Issues  that 
must  be  considered  when  conducting  these  studies  in  the  corporate  en¬ 
vironment  are  considered.  These  include:  (a)  should  corporations  pay 
for  this  type  of  basic  research?  (b)  how  can  the  results  from  Delphi 
studies  be  best  utilized  in  business?  (c)  misusing  Delphi  results  In 
business,  (d)  ln-houae  versus  consultant  conducted  studies,  and  (3)  the 
proprietary  nature  of  business  Delphi  study  results.  The  paper  concludes 
with  some  projections  on  the  future  of  Delphi  in  the  corporate  environ¬ 
ment. 


BEDFORD,  MICHAEL  T. 

"The  Value  of  Competing  Panels  of  Experts  and  the  Impact  of  'Drop-outs' 
on  Delphi  Reeults" 

Delphi:  The  Bell  Canada  Experience ,  Bell  Canada,  October  1972. 

The  primary  objective  of  the  analysis  was  to  determine  whether  the 
division  of  s  Delphi  panel  Into  fairly  distinct  types  of  panelists  would 
increase  the  efficiency  of  the  technique  In  terais  of  information  gener- 
atad.  A  second  objective  was  to  determine  whether  a  number  of  panelists 
dropping  out  of  the  pa**  .-re  the  final  questionnaire  has  a  signifi¬ 

cant  effect  on  the  r  ^ome.  It  was  found  that  there  was  very  little 
difference  between  the  statistical  responses  of  the  two  groups  studied 
(housewives  and  communications  experts)  but  that  the  comments  and  opin¬ 
ions  gensrstsd  were  extremely  valuable  in  developing  an  internally  con¬ 
sistent  view  of  the  future.  The  attempt  to  learn  more  about  the  nature 
of  ti\e  "drop-out"  panelist  was  unsuccessful  due  to  the  small  sample  of 
drop-cuts  In  this  particular  example,  but  a  data  base  In  this  area  has 
been  formed  and  will  be  updated  with  future  study  results. 

BEDFORD,  MICHAEL  T. 

The  Future  of  Communications  Services  into  the  Home 

Bell  Canada  Business  Planning,  September  1972.  (Proprietary) 

This  study  employs  a  two-panel  approach  to  estimate  future  accep¬ 
tance  of  communications  services  In  the  home.  A  panel  of  housewives 
and  a  panel  of  "experts"  prepared  forecaete  and  were  asked  to  resolve 
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differences  between  the  groups'  views  in  a  following  round.  Areas  stud¬ 
ied  included  Electronic  Shopping  from  the  Home,  Remote  Banking,  Elec¬ 
tronic  Home  Security,  Electronic  Programed  Education  in  the  Home.  Ten 
types  of  Information  Retrieval  Services  into  the  Horn  were  also  explored. 
(8) 

BELL  CANADA,  BUSINESS  PLANNING  GROUP 

(Ra.  1105,  620  Belmont,  Montreal  101,  Quebec) 

Sponsor  of  six  Delphi  studies  into  the  future  of  four  subject  areas: 
Education,  Medicine,  Business,  and  Home  Communications .  See  references 
by  authors  Bedford,  Day,  Doyle,  and  Goodwill,  Feldman,  and  Goodwill. 

The  Trans  Canada  Telephone  System  booklet:  "Communications  Computers 
and  Canada"  also  sumnarizes  the  results  from  the  first  three  afore  men¬ 
tioned  studies.  The  Martino  article  in  the  Futurist  (1972),  discusses 
the  Business  Planning  approach  tc  Delphi  research  and  planning  in  the 
corporate  environment.  (B) 

BELL  CANADA 

Lclphi:  The  Bell  Canada  Experience 

October,  1972. 

BENDER,  A.  D. ,  A.  E.  STRACK,  G.  W.  EBRIGHT,  C.  von  HAUNALTER 
"Delphic  Study  Examines  Developments  in  Medicine" 

Futures,  June  1969,  p.  289. 

The  authors  describe  experiences  at  Smith,  Kline  and  French  Labor¬ 
atories  with  the  conduct  of  internal  and  external  Delphi  studies  on  fu¬ 
ture  medical  developments.  Five  areas  of  medicine  development  are  dis¬ 
cussed:  biomedical  research,  diagnosis,  medical  therapy,  health  care, 
and  medical  education.  Results  in  these  areas  are  displayed  graphically. 
A  medical  scenario  of  the  1980's  is  presented.  Comparison  of  medical 
results  from  other  studies  is  shown.  Opinions  on  the  reasons  for  dif¬ 
ferences  between  the  internal  and  external  SKF  panels  are  also  offered. 
(B) 

BERELSON,  B. ,  and  G.  A.  STEINER 

Hunan  Behavior:  An  Inventory  or  Scientific  Findings 

Harcourt,  Brace  and  World,  New  York,  1964. 

BERNSTEIN,  G.  B. 

A  Fifteen-Year  Forecast  of  Information  Processing  Technology 

Research  and  Development  Division,  Naval  Supply  Systems  Command,  January 
20,  1969. 

BERNSTEIN,  G.  B.,  and  M.  J.  CETRON 

"SEER:  A  Delphi  Approach  Applied  to  Information  Processing" 

Techno  Logical  Forecasting,  Vol.  1,  No.  1,  June  1969. 


BJERRUM,  C.  A. 

/ oVec-ts  ■  l.'CH-y.OOU  of  Computer  HcVvlopmente  and  Applications 

Copenhagen:  Parsons  and  Williams,  1968.  Summarized  in  C.  A.  Bjenun, 
"Forecast  of  Computer  Developments  and  Applications:  1968-2000," 

/  :,t ur-.s ,  Vol.  1,  No.  4,  June  1969,  pp.  331-338. 
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BORKO,  H. 

A  Study  of  the  Needs  for  Research  in  Library  and  Information  Science 
Education 

Institute  of  Library  Research,  University  of  California,  Los  Angeles, 
California,  1970. 

BRIGHT,  JAMES  R. 

A  Brief  Introduction  to  Technology  Forecasting :  Concepts  and  Exercises 

Pemaquld  Press,  Austin,  Texas,  1972. 

This  work  book  is  designed  to  be  used  in  conjunction  with  the 
courses  taughc  at  Bright's  Industrial  Management  Center.  Chapter  5, 
"Delphi  Studies  as  an  Aid  to  Corporate  Planning"  is  a  detailed  analysis 
of  the  Delphi  study  conducted  by  Ling-Temco-Vaught  (LTV) .  Corporate 
background  data,  sample  questionnaires,  and  84  study  forecasts  are  il¬ 
lustrated.  The  LTV  Delphi  experience  is  also  used  to  provide  a  data 
base  for  the  Chapter  (II)  on  Cross  Impact  Analysis.  (B) 

BRODEUR,  PAUL 

"Annals  of  Industry — Casualties  of  the  Workplace" 

Ne u  Yorker,  N.  vember  12,  1973. 

BROWN,  B. 

Delphi  Process:  A  Methodology  Used  for  the  Elicitation  of  Opinions  of 
Experts 

The  Rand  Corporation,  P-3925,  September  1968. 

A  description  of  the  Delphi  method  and  some  of  the  areas  to  which 
it  has  and  could  be  applied.  Choosing  the  panel  of  experts,  whose  ex¬ 
pertise  could  be  decided  on  various  grounds,  is  one  problem  of  the 
method;  another  is  the  questioning  technique  itself.  Questionnaires 
have  been  used  in  the  past;  however,  this  method  could  become  cumber¬ 
some.  In  a  few  years  it  should  be  possible  to  equip  each  expert  with 
a  console  for  feeding  responses  to  a  computer,  which  ,/ould  then  compute 
the  group  response  and  feed  back  the  results.  Six  experiments  using 
the  Delphi  method  have  indicated  that  it  may  prove  useful  in  military, 
educational,  and  business  planning,  as  a  tool  for  forecasting  future 
strategic,  economic,  or  other  states.  Other  possible  applications  in¬ 
clude  medical  diagnostics  and  Investment  counseling.  (R) 

BROWN,  fi.,  S.  W.  COCHRAN,  N.  C.  DALftEY 

The  Delphi  Method ,  II:  Structure  of  Experiments 

The  Rand  Corporation,  RM-5957-PR,  June  1969. 

A  compilation  of  the  experimental  designs,  questionnaires  and  re¬ 
sulting  group  response  data  representing  the  raw  materials  of  a  Rand 
evaluation  of  Delphi  procedures.  (Analysis  of  the  data  and  major  conclu¬ 
sions  are  presented  in  RM-58B8.)  The  Delphi  technique  usee  an  anony¬ 
mous  ,  ordarly  program  of  sequential  individual  interrogations,  with  con¬ 
trolled  feedback  from  respondents  between  interrogations,  to  elide  and 
refius  group  Judgments  where  exact  knowledge  is  unavailable.  Ten  ex¬ 
periments  involved  university  students  as  subjects  and  posed  questions 
of  almanac-type  information  having  numerical  answers.  The  overall  aim 
was  to  explore  how  groups  use  Incomplete  information  to  arrive  at  fac¬ 
tual  conclusions.  Different  experiments  tested  different  hypotheses. 
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One  was  designed  to  cosipare  the  relative  accuracy  of  group  answers  ob¬ 
tained  by  the  Delphi  questionnaire-feedback  Method  with  those  obtained 
by  a  structured,  face-to-face  discussion.  (R) 

BROWN.  B. ,  and  0.  HELMER 

Improving  the  Heliability  of  Estimates  Obtained  from  a  Consensus  of 
Experts 

The  Rand  Corporation,  P-2986,  September  1964.  (Also  published  as  ap¬ 
pendix  to  0.  Helner,  Social  Technology ,  New  York,  Basic  Books,  1966.) 

A  report  on  an  experiment  in  the  use  of  expert  opinions.  The  ex¬ 
periment,  involving  the  Delphi  technique  and  the  computation  of  a  con¬ 
sensus  based  on  self-appraised  competence  ratings,  is  described  and  its 
results  analyzed.  (R) 

BUROS,  OSCAR  KR1SEN  (Ed.) 

The  Sixth  Mental  Measurements  Yearbook 

Cryphon  Press,  New  Jersey,  1365. 

CAMPBELL,  R.  M. 

"The  Delphi  Technique:  Implementation  in  the  Corporate  Environment" 
Management  Services,  Vol.  5,  No.  6,  November-December  1968,  pp.  37-4?. 

This  article  investigates  the  possibilities  of  using  Delphi  in  the 
formal  corporate  environment,  and  points  out  some  of  the  techniques  in 
this  context.  The  method  is  explained  in  the  more  or  less  standard 
manner,  and  the  importance  of  panel  selection  and  the  need  for  some 
structure  to  evaluate  and  use  the  results  are  stressed.  Its  use  is 
proposed  for  the  development  of  new  products.  The  article  is  brief 
and  general.  (P) 

CAMPBELL,  R.  M. 

A  Methodological  Study  of  the  Utilization  of  Experts  in  Business  Fore- 
cas ti  ng 

Ph.D.  Thesis,  University  of  California,  Los  Angeles,  1966. 

This  thesis  examines  current  (1966)  business  forecasting  techniques, 
the  forms  of  group  forecasting,  and  develops  a  number  of  hypotheses  re¬ 
lated  to  the  use  of  Delphi.  An  experimental  set  of  Delphi  studies  is 
developed  and  conducted  to  test  the  hypotheses.  The  forecast  data  are 
16  economic  and  business  statistical  series.  After  analysis,  the  author 
draws  a  number  of  conclusions  on  the  Delphi  technique.  He  also  suggests 
a  number  of  marketing  applications  for  the  Delphi  process.  (B) 

CANADIAN  COMPUTER /COMMUNICATIONS  TASK  FORCE 

Branching  Out,  Vol.  2 

Information  Canada,  Ottawa,  May  1972. 

The  task  force  report  exastines  various  policy  options  for  the 
Canadian  government  in  the  computer  cni— unlcationa  field.  The  second 
voltuse  contains  various  appendices.  The  appendix  on  Education  applica¬ 
tions  lntsgrstes  the  results  of  the  Bell  ciuada  Education  Delphi  study 
with  other  material  (pp.  101-129).  (B) 
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CANTRIL,  HADLEY 

"The  Prediction  of  Social  Events” 

Journal  of  Abnormal  and  Social  Psychology,  Vol.  33,  1938,  pp.  364-389. 
CARSON,  ROBERT 

Interaction  Concepts  of  Personality 

Aidine  Publishing  Co.,  Chicago,  1969. 

CARTWRIGHT,  D. ,  and  A.  ZANDER 

Crony  Dynamics 

Row  Peterson,  Evanston,  Illinois,  1960. 

CETRON,  M. ,  and  D.  OVERLY 
"Toward  a  Consensus  on  Che  Future" 

Innovation,  No.  31,  New  York,  Hay  1972. 

This  study  is  an  Interesting  one,  even  though  it  is  more  of  a 
survey  than  a  Delphi  study.  The  authors  have  gone  through  their  data 
bank  of  forecasts,  many  from  Delphi  studies,  and  prepared  a  question¬ 
naire  on  74  future  events  in  two  broad  categories:  "general"  events 
and  "technical"  events.  The  first  category,  business,  political,  and 
social  changes,  has  a  time  frame  to  1980.  The  second  category  goes  to 
1980  and  1985.  The  respondent  will  be  anyone  who  tears  out  the  ques¬ 
tionnaire  from  the  journal  and  returns  it  to  the  authors.  Since  the 
journal  ip  business  oriented,  we  can  expect  that  a  large  number  of  bus¬ 
inessmen  will  respond.  Answers  to  be  published  in  the  future.  (B) 

CETRON.  M.,  and  C.  RALPH 

Industrial  Applications  of  Technological  Forecasting 

Wiley-lnteracience ,  Toronto,  1971. 

This  book  is  designed  to  stress  the  practical  applications  of  T/v 
In  Industry.  The  use  of  Delphi  by  several  corporations  is  outlined 
(often  unnamed).  These  firms  include:  "A  Petrochemicals  Research  In¬ 
stitute,"  "Man-Made  Fibers  Co.,"  (material  shown),  Monsanto  Corp., 

Smith,  Kline  and  French  (examples),  LTV  (examples),  "Large  Copper  Com¬ 
pany."  (B) 

COCHRAN,  WILLIAM  G. 

Sampling  Techniques 

Wiley,  New  York,  1963. 

"Computers  in  the  Crystal  Ball" 

Science  Journal,  August  1969,  p.  15. 

Short  reference  to  the  Parsons  and  Williams  Delphi.  Two  charts 
from  the  study  showing  projections  on  computer  applications  and  computer 
development  are  shown.  (B) 

CON-FORM 

See  study  described  in  the  LACHHANN  article. 

CURRILL ,  D.  L. 

"Technological  Forecasting  in  Six  Major  U.K.  Companies" 

Long  Range  Planning,  March  1972. 
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Describea  a  survey  by  the  author  of  100  U.K.  companies  (37  respon¬ 
dents)  on  technological  forecasting.  Techniques  used  by  six  companies 
are  described.  Delphi  is  used  by  a  "Glass"  Company,  a  "Consumer  Goods" 
Company,  2  "Chemical"  Companies,  and  an  "Electrical  Engineering"  Company. 
Study  results  not  shown,  but  the  importance  of  various  techniques  in 
the  companies  is  described.  (B) 

DALKEY,  N.  C. 

"Analyses  from  a  Group  Opinion  Study" 

Futures,  Vol.  1,  No.  6,  December  1969,  pp.  541-551. 

DALKEY,  N.  C. 

The  Delphi  Method:  An  Experimen tal  Study  of  Group  Opinion 

The  Rand  Corporation,  RM-5888-PR,  June  1969.  (See  also  RM-5957,  P-2983, 
and  P-3721.) 

A  report  of  results  of  experimentation  on  the  effectiveness  of  Del¬ 
phi  procedure,  which  Incorporate  anonymous  response,  iteration  and  con¬ 
trolled  feedback,  and  statistical  group  response  to  elicit  and  refine 
group  Judgments  where  exact  knowledge  is  unavailable.  In  spring  1968, 
Hand  conducted  ten  experiments  using  over  150  university  students. 
Questions  related  to  almanac-type  information.  Results  showed  that  con¬ 
trolled  feedback,  compared  with  face-to-face  discussion,  Improved  the 
accuracy  of  group  estimates,  thus  validating  the  use  of  Delphi  tech¬ 
niques  In  areas  of  partial  information.  Insight  was  gained  into  group 
informa:lon  processes.  A  meaningful  estimate  of  the  accuracy  of  a  group 
response  to  a  given  question  can  he  obtained  by  combining  individual 
self-rating  of  competence  on  that  question  into  a  group  rating.  Adding 
this  result  to  an  observed  relationship  between  accuracy  and  standard 
deviation  makes  it  possible  to  attach  accuracy  scores  to  the  products 
of  a  Delphi  exercise.  (R) 

DALKEY,  N.  C. 

quality  of  Life 

The  Rand  Corporation,  P-3805,  March  1968. 

DALKEY,  N.  C. 

Frediotiny  the  Future 

The  Rand  Corporation,  P-3948,  October  1968. 

Opinion  is  basic  to  long-range  developmental  forecasting.  The 
difficulties  (such  as  the  influence  of  dominant  Individuals,  noise,  and 
group  pressure  for  conformity)  of  obtaining  a  group  opinion  through 
traditional  face-to-face  interaction  led  to  the  development  of  the  Del¬ 
phi  procedures,  which  are  described  in  this  paper.  The  characteristics 
of  these  procedures — anonymity,  iteration  with  controlled  feedback,  and 
statictical  group  response — derive  from  Rand's  discovery  that  simply 
the  average  of  individual  opinions,  without  discussion,  tends  to  be 
more  accurate  than  group  opinion  resulting  from  discussion.  The  exper¬ 
iments  that  led  to  these  results  Involved  almanac  questions,  such  as. 

How  many  votes  did  Kennedy  receive  in  the  i960  Presidential  election 
in  Texas?  An  initially  vide  range  of  answers  were  found  to  gradually 
converge,  improving  in  accuracy  in  the  majority  of  cases;  the  pattern 
of  responses  resembled  a  log-normal  curve.  Further  studies  will  attempt 
to  dampen  the  effect  of  group  pressure  while  amplifying  accuracy.  (R) 
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DALKEY,  N.  C. 

Experiments  in  Group  Prediction 

The  Rend  Corporation,  P-3820,  March  1968. 

The  uae  of  the  Delphi  method  for  group  prediction  and  estimating 
in  a  series  of  Rand  experiments.  The  method  has  three  basic  features: 

(1)  It  elicits  individual  opinion,  usually  by  questionnaire,  but  opin¬ 
ions  are  not  attributed  to  specific  individuals  when  communicated  to 
the  group.  (2)  It  provides  controlled  feedback:  An  exercise  is  con¬ 
ducted  in  several  rounds,  opinions  generated  during  one  round  being  fed 
back  to  the  group  on  the  next  round,  usually  in  the  form  of  statistical 
summaries.  (3)  Group  opinion  is  expressed  in  terms  of  a  statistical 
score.  In  most  cases,  there  is  a  pronounced  convergence  of  opinion  with 
iteration;  a  wide  spread  on  the  initial  round  decreases  monotonlcally 
on  succeeding  rounds,  principally  between  the  first  and  second.  Where 
accuracy  of  response  can  be  checked,  it  is  shown  to  Increase  with  iter¬ 
ation.  Recent  Rand  experiments  have  focused  on  the  use  of  information 
that  can  be  readily  verified  as  a  means  of  further  investigating  the 
efficacy  of  the  Delphi  technique.  (R) 

DALKEY,  N.  C. 

Delphi 

The  Rand  Corporation,  P-3704,  October  1967.  (Sae  also  P-3558.) 

An  outline  of  the  Delphi  technique  of  long-range  forecasting  by 
separately  eliciting  and  refining  the  opinions  of  a  group  of  advisors 
without  contact  among  them,  and  calculating  a  statistical  "group  re¬ 
sponse."  The  procedure  was  designed  to  overcome  the  disadvantages  com¬ 
mon  to  committees  and  small  groups.  The  experts  reply  to  written  ques¬ 
tionnaires  or  an  online  computer,  receive  statistical  feedback  through 
forms!  lines  of  communication,  and  raaubmlt  their  estimates.  Where  the 
response  is  a  number  (such  as  a  date  or  amount),  the  most  useful  index 
has  been  the  median  of  the  individual  estimates.  During  the  process, 
opinions  do  converge;  where  answers  can  be  checked  ageinst  reality,  it 
is  found  that  the  median  response  tends  to  move  in  the  direction  of  the 
true  answer.  Self-confidence  is  not  correlated  with  Individual  perfor¬ 
mance,  but  the  subgroup  with  the  highest  self-ratings  for  competence 
will  consistently  perform  slightly  better  than  the  group  as  a  whole. 

(R) 

DALKEY,  N.  C.,  end  B.  BROWN 

Comparison  of  Group  Judgment  Techniques  with  Short-Hange  Predictions 
and  Almanac  Questions 

The  Rand  Corporation,  R-678-ARPA,  May  1971. 

An  experiment  designed  to  discover  whether  the  results  of  labora¬ 
tory  studies  dealing  with  general  (almanac)  Information  are  relevant 
to  the  applied  case  vhan  the  true  answer  la  unknown.  Using  short-range 
prediction  questions  aa  subject  matter,  the  experiment  indicates  that, 
in  general ,  Delphi  procedures  ere  at  least  as  effective  with  short-range 
prediction  as  thsy  hsvs  bean  for  almanac  material.  Eight  groups,  of 
about  20  each,  of  upper-classmen  and  college  graduates  were  given  short - 
range  prediction  questions  to  answer  in  a  two-round  Delphi  exercise. 
Satisfactory  answers  wars  obtained  for  32  of  the  40  questions.  Correl¬ 
ations  betwsen  standard  deviation  end  accuracy,  and  between  group  self- 
reting  and  accuracy,  were  significantly  higher  for  the  prediction  than 


for  Che  almanac  question*.  Half  Che  groups  generated  estimates  of  the 
three  quartiles  of  the  distribution;  Che  ocher  half  generated  point  es¬ 
timates.  No  significant  difference  was  observed  between  these  two  kinds 
of  estimates.  (R) 

DALKEY,  N.  C. ,  B.  BROWN,  and  S.  W.  COCHRAN 

The  Delphi  Method ,  IV:  Effect  of  Percentile  F'eedl  ok  and  Feed-In  of 
Relevant  Pacts 

The  Rand  Corporation,  RM-6118-PR,  March  1970. 

An  investigaCion  of  Che  effect  on  group  accuracy  of  two  variations 
in  the  Delphi  procedures.  In  these  exercises,  twenty  general  informa¬ 
tion  questions  are  answered  by  two  groups  of  respondents,  who,  after 
receiving  some  form  of  feedback,  may  revise  their  answers.  In  the  first 
variation,  feeding  back  individual  percentiles  resulted  in  no  improve¬ 
ment  over  feeding  back  the  median  and  quartiles  of  the  group  response. 

On  the  ocher  hand,  in  the  second  variation,  adding  a  relevant  fact  to 
the  median  and  quartiles  information  resulted  in  a  statistically  sig¬ 
nificant  increase  in  numerical  accuracy.  The  number  of  changed  an¬ 
swers  was  also  greater,  suggesting  that  introduction  of  a  relevant  fact 
strengthens  motivation  fo-  revision.  For  a  number  of  military  concerns, 
such  os  long-range  technological  development  assessment  or  future  threat 
evaluation,  the  expertise  of  a  group  of  decisionmakers  is  relied  on. 

The  Delphi  studies  are  an  effort  to  Improve  suc^  judgments  through  re¬ 
fined  procedures.  (R) 

DALKEY,  N.  C. ,  B.  BROWN,  and  S.  W.  COCHRAN 

The  Delphi  Method  III:  Uee  of  Self-Ratings  to  Improve  Group  Estimates 

The  Rand  Corporation,  RM-6115-PR,  November  1969. 

An  analysis  of  the  validity  of  using  self-ratings  as  a  technique 
for  selecting  more  accurate  subgroups  In  applications  of  the  Delphi 
procedures  for  eliciting  group  judgments.  A  series  of  experiments  was 
conducted  using  sixteen  groups  of  upper-class  and  graduate  college  stu¬ 
dents  answering  almanac- type  questions  (twenty  subjects  per  group  and 
twenty  questions  per  subject).  The  findings  indicate  that  If  the  dif¬ 
ference  in  average  suif-reting  between  the  subgroups  is  substantial, 
and  if  the  subgroups  are  held  to  reasonable  size,  both  the  degree  of 
Improvement  and  the  total  number  of  improvements  are  greater  than  when 
feedback  alone  is  used.  This  study  augments  the  results  reported  in 
RM-5888  and  RM-5557.  (R) 

DALKEY,  M.  C. ,  and  0.  HELMER 

An  Experimental  Application  of  the  Delphi  Method  to  the  Use  of  Experts 

The  Rand  Corporation,  RM-727-PR  (Abridged),  July  1962.  (Also  published 
in  Management  Science ,  Vol.  9,  No.  3,  April  1963,  pp.  458-467.) 

An  abridgment  and  revision  of  RM-727,  The  Use  of  Experts  for  the 
Estimation  of  Bombing  Requirements:  A  Project  Delphi  Experiment . 

DALKEY,  N.  C. ,  R.  J.  LEWIS,  end  D.  SNYDER 

Measurement  and  Analysis  of  the  Quality  of  Life:  With  Exploratory  Il¬ 
lustrations  of  Applications  to  Career  and  Transportation  Choices 

The  Rand  Corporation,  RM-6228-DOT,  August  1970. 
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Ths  future  of  transportation  and  its  impact  on  regional  environment 
depends  in  part  on  the  interaction  with  other  programs  such  as  housing, 
education,  health  and  welfare.  This  memorandum  summarizes  attempts  thus 
far  to  determine  whether  it  is  feasible  to  Identify  and  measure  under* 
lying  dimensions  of  the  quality  of  life  (QOL)  and  relate  them  to  trans¬ 
portation  findings.  A  model  of  individual  QOL  includes  a  set  of  general 
qualities  of  the  stream  of  events  occurring  to  an  individual  that  largely 
determine  his  sense  of  well-being.  Several  group  judgment  (Delphi) 
studies  produced  relatively  well-defined  lists  of  such  qualities,  in¬ 
cluding  self-respect,  affection,  security,  health,  achievement,  novelty, 
freedom,  comfort  and  aggression.  Other  studies  compared  lists  of  qual¬ 
ities  with  employment-environment  opportunities  and  transportation 
choices.  A  number  of  approaches  are  suggested  for  future  investigation, 
for  example,  in-depth  time-event  studies  with  small  groups  and  cross- 
sectional  national  surveys  based  on  a  QOL  model.  (R) 

DALKEY,  N.  C.,  and  D.  L.  ROURKLE 

Experimental  Assessment  of  Delphi  Procedures  u'ith  Croup  Value  , Judgments 

The  Rand  Corporation,  R-612-ARPA,  February  1971.  (See  also  RM-5888, 
RM-5957,  RM-6118. ) 

One  of  a  series  of  studies  using  Delphi  procedures  to  aid  decision¬ 
makers  in  dealing  wltn  value  judgments.  Previous  studies  have  not 
clearly  shown  that  there  is  an  appropriate  population  of  factual  ques¬ 
tions  to  compare  with  value  judgments;  the  variability  of  performance 
on  factual  questions  in  large,  depending  on  the  type  of  questions  asked. 
Nith  this  in  mind,  some  comparisons  were  made:  Two  groups  of  UCLA  stu¬ 
dents  were  asked  to  generate  and  rate  lists  of  value  categories  that 
they  considered  important  to  higher  education  and  the  quality  of  life. 
Analyses  showed  that  (1)  distributions  were  generally  single-peaked  and 
roughly  bell-shaped,  (2)  the  correlations  between  different  groups  and 
different  rating  methods  were  high,  and  (3)  the  number  of  changes  and 
degree  of  convergence  for  vslue  judgments  (reduction  in  standard  devia¬ 
tion)  were  comparable  to  similar  indices  for  factual  judgments.  The 
experiment  supported  the  conclusion  that  Delphi  procedures  are  appro¬ 
priate  for  processing  value  material  as  well  as  factual  material.  (R) 

DALKEY,  N.  C. ,  D.  L.  ROURKE,  R.  LEWIS,  and  D.  SNYDER 
Studies  in  the  Quality  of  Lift:  Delphi  and  Decision  Making 

D.  D.  Heath,  Lexington,  Mass.,  1972. 

DARI All,  JEAN -CLAUDE,  and  FRANCOISE  M0R1ZE 

"Delphi  in  the  Assessment  of  Research  and  Development  Projects" 

Futures  ,  October  1973,  pp.  469-483. 

DAVIS,  RICHARD  C. 

"Organising  and  Conducting  Technological  Forecasting  in  a  Consumer 
Goods  Firs" 

In  Jamas  R.  Bright  and  M.  E.  F.  Schosman  (eda.),  A  Guide  to  Practical 
Technological  Forecasting ,  Prsntlcs  Hall,  Inc.,  Englawood  Cllffa,  New 
Jaraay,  1973. 

The  uaa  of  technological  forecasting  techniques  for  e  consumer 
goods  firm's  product  planning  procedures  is  described.  In  addition  to 


describing  Whirlpool's  information  storage  and  retrieval  system,  the 
author  illustrates  how  Delphi  has  been  used  as  an  information  gathering 
technique.  Parc  of  a  sample  questionnaire  is  illustrated.  (B) 

DAY,  LAWRENCE  H. 

"The  Future  of  Computer  and  Communications  Services" 

National  Computer  Conference  and  Exposition,  New  York,  June  1973. 

The  future  applications  of  computer  and  communications  capabilities 
in  the  three  application  areas  are  described  (education,  business-travel 
consnunications  tradeoffs,  and  the  home).  The  projections  discussed  have 
been  drawn  from  a  number  of  Delphi  studies — Bell  Canada,  Parsons  &  Wil¬ 
liams,  I.F.F.  work,  EDUCOM.  Comparisons  between  the  various  forecasts 
are  discussed.  (B) 

DAY,  LAWRENCE  H. 

"Long  Term  Planning  In  Bell  Canada" 

.l.inmal  of  Lon<j  i\‘a>uje  Planni nj ,  London,  1973.  (Submitted  for  publica¬ 
tion.) 

This  paper  describes  the  Business  Planning  process  at  Bell  Canada . 
The  use  of  many  technological  forecasting  techniques  in  gathering  in¬ 
put  data  is  described.  The  use  of  Delphi  as  a  part  of  this  gathering 
of  "futures”  information  is  discussed.  Sample  Delphi  questionnaires 
and  results  are  illustrated.  (B) 

DAY,  LAWRENCE  H. 

"Delphi  Research  in  the  Corporate  Environment" 

Delphi:  The  bell  Canada  Expert, :nc>  ,  Bell  Canada,  October  1972. 

This  article  examines  the  use  of  the  Delphi  technique  in  the  cor¬ 
porate  environment.  The  paper  indicates  that  a  broad  range  of  Delphi 
studies  have  been  conducted  or  sponsored  by  various  corporations  In 
North  America  and  Europe.  These  studies  were  conducted  by  consultants 
or  planning  groups  in  ths  business  firms.  Four  Delphi  studies  conducted 
by  Bell  Canada's  Business  Planning  Group  are  examined.  Results  from 
these  studies  are  illustrated.  Several  Important  Issues  that  must  be 
considered  when  conducting  these  studies  in  the  corporate  environment 
are  considered.  These  Include:  (a)  should  corporations  pay  for  this 
type  of  basic  research?  (b)  how  can  the  results  from  Delphi  studies 
be  beat  utilized  in  business?  (c)  misusing  Dslphi  results  in  business, 
(d)  in-house  versus  consultant  conducted  studies,  and  (e)  the  propri¬ 
etary  nature  of  business  Delphi  study  rssults.  Ths  paper  concludes 
with  some  projections  on  the  future  of  Delphi  in  the  corporate  environ¬ 
ment  . 

DEAN,  B.  V.,  and  S.  MATHIS 

Anilysic.  of  the  Exploratory  Development  Project  Evaluation  Experiment 

Department  of  Operations  Research,  Casa  Western  Reserve  University, 
Technical  Memorandum  No.  165,  1969. 

This  report  describes  e  study  done  for  the  Army  Meteriel  Command 
using  modified  Delphi  procedure  to  evaluate  research  and  development 
projects,  whose  value  is  of  course  often  of  s  very  subjective  nature. 
Data  were  collected,  using  a  modified  Delphi,  from  a  twelve  member 
panel  and  linear  regression  models  of  project  value  were  constructed 
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48  a  function  of  eight  critical  factors.  There  is  a  description  of  the 
procedures  used  and  conclusions  drawn.  Delphi  has  its  usual  attendant 
advantages,  disadvantages  and  questions;  the  addition  of  regression  ana¬ 
lysis  aids  in  the  evaluation  of  reaults.  (P) 

DE  BRIGARD,  R. ,  and  O.  HELMUT. 

Some  Potential  Developments:  1970-2000 

Institute  for  the  Future,  R-7,  April  1970. 

DELPHI  PANEL  ON  THE  FU'URE  OF  LEISURE  AND  RECREATION,  SET  INC. 

Los  Angeles,  1972.  (Multiclient  Proprietary  Study) 

This  study  was  conducted  by  SET,  Inc.  (Social  Engineering  Technol¬ 
ogy)  for  a  group  of  clients  interested  in  market  opportunities  resulting 
from  Increased  leisure.  (B) 

DERIAN,  JEAN-CLAUDE,  and  FRANCOISE  MORIZE 

"Delphi  in  the  Assessment  of  Research  and  Development  Projects" 

Futures,  October  1973,  pp.  469-483. 

DICKSON,  PAUL 

Think  Tanks 

Atheneum,  New  York,  1971. 

This  Journalistic  history  and  study  of  the  growth  of  "think  tanks" 
in  the  U.S.  is  mainly  concerned  with  developments  in  the  governmental 
field.  The  discussion  of  Delphi  forecasting  (pp.  313-18  and  pp.  324-36) 
does  mention  that:  "Delphi  is  now  coming  into  widespread  use  in  indus¬ 
try  to  determine  new  markets,  possible  new  products  for  the  future,  and 
pitfalls  to  development.  No  detailed  examples  are  given  except  for  a 
reference  to  the  TRW  work.  (B) 

DOLE.  S.  H.,  et  al 

Establishment  of  a  Long-Range  Planning  Capability 

The  Rand  Corporation,  RM-6151-NASA,  September  1969. 

An  examination  of  some  of  the  major  problems  of  performing  an  ef¬ 
fective  long-range  planning  function  within  NASA  and  a  survey  of  some 
of  the  techniques  of  systems  analysis  that  might  aid  in  the  task  of 
overall  agency  planning.  Long-range  objectives  and  policies  are  defined 
and  developed,  and  the  consequences  of  future  decisions  analyzed,  by 
structuring  the  planning  process  into  five  procedural  phases:  input, 
projection,  creative,  analytical,  and  output.  Concurrent  supporting 
analysis  is  used  to  develop  an  Information  base  for  decisionmaking  on 
alternative  strategies.  In  this  context,  the  major  and  some  minor  tech¬ 
niques  of  modem  systematic  analysis  are  surveyed  to  determine  their 
applicability.  Those  clearly  applicable  are:  (1)  many  classical  sys¬ 
tems  analysis  methods,  and  methods  for  coping  with  uncertainty;  (2) 
Delphi  procedures;  (3)  worth  assessment  techniques;  (4)  relevance  treea 
end  morphological  analysis;  (5)  other  forecasting  techniques.  The  re¬ 
maining  approaches  are  either  uncertain  or  clearly  inapplicable.  A 
long-range  planning  function  would  significantly  aid  the  overall  NASA 
program,  but  it  would  require  continuing  support  by  top  management  and 
coordination  with  related  planning  areas.  (R) 
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DOLE,  S.  H. ,  et  al. 

Methodologies  for  Analyzing  the  Comparative  Effectiveness  and  Costs  of 
Alternative  Space  Plane:  Volume  1  (Suimary  Volume) 

The  Rand  Corporation,  RM-5656-NASA,  August  1968.  (Limited  distribution.) 
DOLE,  ~  H.,  et  al. 

Methodologies  for  Analyzing  the  Comparative  Effectiveness  and  Costs  of 
Alternative  Space  Plans:  Volume  2  (Technical  Volume) 

The  Rand  Corporation,  RM-5656-NASA,  August  1968.  (Limited  distribution. ) 
DOYLE,  FRANK  J.,  and  DANIEL  Z.  GOODWILL 

"An  Exploration  of  the  Future  in  Educational  Technology"  in  H.  A. 
Stevenson,  R.  M.  Stamp,  and  J.  D.  Wilson,  (eds.),  The  Best  of  Times 
The  Worst  of  Tines — Contemporary  Issues  in  Canadian  Education  Holt, 
Rinehart,  and  Winston,  Montreal,  1972. 

The  material  from  this  article  is  extracted  from  the  Bell  Canada 
Educational  Delphi.  Areas  covered  include  changing  values  of  society, 
general  trends  in  terminal  use,  and  the  acceptance  of  computerized  li¬ 
brary  systems,  computer  aides  Instruction  systems,  and  visual  display 
systems.  (B) 

DOYLE,  FRANK  J.,  and  DaNIEL  Z.  GOODWILL 
An  Exploration  of  the  Future  in  Educational  Technology 

Bell  Canada  Business  Planning,  Montreal,  January  1971.  (Proprietary) 
This  study  explores  future  acceptance  of  a  number  of  visual  ^nd 
computer  communications  services:  Computer  assisted  instruction,  com¬ 
puterized  library  systems,  and  audio  visual  display  systems.  Other 
areas  explored  include  value  trends  (1970-2000),  chemical  learning, 
evolution  in  school  design,  changing  role  of  the  teacher,  and  technology 
in  the  home.  (B) 

DOYI^I ,  FRANK  J.,  and  DANIEL  Z.  GOODWILL 
An  Exploration  of  the  Future  in  Medical  Technology 

Bell  Canada  Business  Planning,  Montreal,  March  1971.  (Propt  etary) 

This  study  explores  future  acceptance  of  services  such  as  multi- 
phasic  screening.  Computer  assisted  diagnosis,  computerized  medical 
libraries  and  terminal  capabilities  in  medicine.  Other  areas  explored 
included  value  trends,  remote  physiological  monitoring,  future  technol¬ 
ogy  in  the  home,  and  changing  roles  in  the  medical  profession.  (B) 

DROR,  Y. 

"La  Prediccion  de  lo  Politicamente  Posible" 

Hevieta  Espcmola  de  la  Opinion  Publica,  July  -  December  1970,  pp.  21-22, 
89-98.  (See  also  The  Rand  Corporation,  P-4044,  April  1969.) 

A  reprint  from  Futures,  issue  not  specified.  The  application  of 
the  Delphi  method  to  factual  political  prognosis  is  considered.  Factual 
political  forecasting,  contrary  to  politically  oriented  predictions, 
involves  reference  to  an  agent,  an  alternative  policy  and  a  political 
area.  It  can  be  approached  via  the  following  variables:  principal 
agents,  their  capacities  and  intentions;  actual  and  potential  forces 
within  the  political  area;  the  interactions  between  agent  and  increased 
political  leverage;  the  critical  influence  of  the  masses.  A  three-fold 
theoretical  schema  baaed  on  the  Delphi  method  is  presented  for  political 
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forecasting.  For  a  ‘.alia  on  Lite  Delphi  method,  see  0.  Helmer,  Social 
Technology ,  New  York,  New  York,  Basic  Books,  1966.  The  three  parts  of 
the  schema  are:  (a)  direct  estimates  of  political  practicability,  (b) 
conditions  of  political  practicability  and  (c)  estimates  of  the  vari¬ 
ables  of  political  practicability.  This  method  allows  for  evaluating 
the  strength  of  a  political  possibility  concerning  each  political  al¬ 
ternative.  A  side  benefit  is  the  education  of  experts  in  prediction 
in  general  and  in  political  analysis  in  particular.  A  word  of  warning 
is  given  concerning  the  capacity  of  unpredictable  human  forces  to  over¬ 
come  apparently  unsurmountable  barriers.  This  Introduces  an  element  of 
the  provisional  into  all  political  forecasting. 

ENZER,  SELVYN 

Some  Development  in  Plastics  and  Competing  Materials  by  1985 

Institute  for  the  Future,  R-17,  Jaruary  1971. 

Owens- "orning  sponsored  this  study  which  attempted  "to  focus  upon 
possible  coibinatlons  of  material  property  changes  that  are  likely  to 
affect  widespread  material  usage."  The  materials  considered  by  the 
Delphi  panel  included:  engineering  plastics,  general  purpose  and  spe¬ 
cialty  plastics,  glass  fiber  reinforced  plastics,  foamed  plastics,  and 
nonplastics.  The  study  includes  forecasts  of  U.S.  plastic  production, 
anticipated  changes  in  properties  of  existing  materials  and  developments 
in  other  important  materials  to  1985.  (B) 

ENZER,  SELWYN 

Some  Prcspcctc  for  Residential  Housing  by  1385 

Institute  for  the  Future,  R-13,  January  1971. 

Owens-Corning  Flberglas  sponsored  this  study  to  determine  the  moat 
probable  trends  in  residential  housing  in  the  coming  15  years  and  devel¬ 
opments,  actions,  and  policies  that  could  alter  probable  trends.  The 
panel  results  Include  forecasts  of  housing  supply  and  demand,  housing 
costs,  and  institutions  and  monetary  aspects  of  housing,  housing  tenure, 
building  codes,  technological  developments  affecting  residential  housing 
and  society.  Panel  charts  and  comments  are  included.  (B) 

ENZER,  SELWYN 

Delphi  and  Cross-Impact  Techniques :  An  Effective  Combination  for  Sys¬ 
tematic  Futures  Analysis 

Institute  for  the  Future,  WP-8,  June  1970. 

ENZER,  SELWYN 

A  Case  Study  Using  Forecasting  as  a  Decistonmaking  Aid 

Institute  for  the  Future,  WP-2,  December  1969. 

ENZER,  SELWYN,  and  R.  DE  BRIGARD 

Issues  and  Opportunities  in  the  State  of  Connecticut;  1970-2000 

Institute  for  the  Future,  R-8,  March  1970. 

"The  Exploration  of  tha  Future" 

RialitSs,  No.  245,  June  1966,  pp.  50-58.  Translated  from  the  French  by 
R.  Nclswender,  The  Rand  Corporation,  P-3540,  February  1967. 
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Translation  from  the  French  of  an  article  in  R^alit&s  on  the  world 
of  the  future  as  visualized  by  participants  in  an  international  confer¬ 
ence  organized  by  R£alitds.  The  article  reports  the  work  of  a  group  of 
specialists  whose  interests  are  focused  on  a  systematic  exploration  of 
the  future,  their  opinions  on  the  advantages  and  disadvantages  of  life 
in  the  21st  century,  and  their  varying  approaches  to  predicting  and 
planning  for  the  future.  Among  the  forecasting  methods  described  is 
the  Delphi  technique.  In  the  opinion  of  R6alit6a,  the  time  has  come  to 
establish  both  national  and  international  institutes  for  prediction  and 
planning.  (R) 

ENZER,  SELWYN,  DENNIS  LITTLE,  and  F.  D.  LAZAR 

Some  Prospects  for  Social  Change  by  198S  and  Their  Impact  on  Time /Money 
Budgets 

Institute  for  the  Future,  R-25,  March  1972. 

General  Telephone  and  Electronics  sponsored  this  study.  The  re¬ 
port  outlines  potential  developments  and  trends  likely  to  produce  major 
changes  in  patterns  of  time/money  expenditures  in  the  next  fifteen  years. 
The  study  also  had  a  secondary  objective  of  experimenting  with  social 
forecasters  using  a  modified  Delphi  approach.  (B) 

ESSO  (EXXON) 

Participants  in  a  study  and  users  of  Delphi  output  in  multi-indus¬ 
try  study  described  by  GLAZIER,  et  al .  (B) 

FARQUHAR,  J.  A. 

A  Preliminary  Inquiry  into  the  Software  Estimation  Process 

The  Rand  Corporation,  RM-6271-PR,  August  1970. 

Reviews  the  literature  of  software  estimation  and  reports  a  small 
experiment  comparing  Delphi  with  face-to-face  group  judgment  to  predict 
the  time  necessary  to  program  an  information  system — in  this  case,  the 
Air  Forces's  PDSO  (Personnel  Data  System — Officers).  Planning  software 
production  is  necessary  but  almost  impossible  at  present.  Cost  to  com¬ 
pletion  depends  on  many  factors,  some  unknown  at  the  time  and  all  hard 
to  quantify  (the  difficulty  of  the  task,  the  programmer's  ability  and 
familiarity  with  the  procedures  involved,  the  degree  of  definition  pro¬ 
vided  him,  and  about  80  other  factors).  The  experiments  undertaken 
failed  to  establish  the  utility  of  either  estimation  method.  Primary 
recommendations  for  further  research  are:  (1)  more  effective  data  col¬ 
lection,  (2)  analysis  of  characteristics  of  good  estimators,  and  (3) 
formal  inquiry  into  the  techniques  used  by  estimators.  (R) 

FELDMAN,  N.  E. 

"Communication  Satellite  Output  Devices" 

! h  Microwave  Journal,  Nov.  and  Dec.  1965  issues. 

FELDMAN,  PHILIP 

"Internal  and  External  Delphi  Panel  Comparison" 

...  h ■: :  ;h>  he  l'  'letl;  Expert .■>,*  ,  Bell  Canada,  October  19  72. 

To  determine  whether  there  are  significant  differences  in  the 
responses  of  Internal  and  external  Delphi  panels  to  Identical  question¬ 
naires.  To  determine  vhether  it  will  be  necessary  to  go  outside  the 
company  for  panelists  on  future  Delphi  studies. 
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FELDMAN,  PHILIP 

A  Technology  Assessment  of  Computer-Assisted  Instruction 

Bell  Canada,  Business  Planning,  August  1972  (Internal  document). 

This  study  has  two  main  objectives:  (a)  to  illustrate  a  methodol¬ 
ogy  that  will  be  of  use  for  technology  assessments,  (b)  to  have  a  pilot 
assessment  of  the  societal  impacts  of  computer  assisted  instruction  in 
post  secondary  institutions.  Feldman  uses  input  from  the  Bell  Canada 
Educational  and  Business  Delphi  studies  in  the  assessment.  The  report 
illustrates  how  Delphi  study  results  can  be  used  as  means  towards  ends 
rather  than  ends  in  themselves.  (B) 

FISHBEIN,  M.  (Ed.) 

Readings  in  Attitude  Theory  and  Measurement,  Wiley,  New  York,  1967. 

"Forecasters  Turn  to  Group  Guesswork:  Delphi  Technique  is  Catching  on 
with  Corporations" 

Business  Week,  No.  2115,  March  1970,  pp.  130-34. 

This  article  describes  recent  expansion  of  the  Rand,  Institute  for 
the  Future,  and  National  Industrial  Conference  Board  activities.  Cor¬ 
porate  activities  described  include  McDonald  Douglas,  Weyerhaeuser  Co., 
Smith,  Kline  and  French,  and  TRW.  Most  references  are  brief  and  the 
article  is  mainly  survey  in  nature.  The  reser  ch  of  Dalkey  at  Rand  is 
also  discussed.  (B) 

GENERAL  DYNAMICS,  CONVAIR  DIVISION 

Discussed  in  Martino  Futurist  (1972)  article.  Three  studies  are 
mentioned.  The  Aerospace  group  organized  a  Delphi  study  on  the  future 
of  fluidics  to  guide  Convair  R&D  in  the  field.  The  second  study  ex¬ 
amined  the  relative  values  of  possible  non-destructive  testing  tech¬ 
niques.  This  data  was  used  for  corporate  decisions  on  the  value  of 
composite  materials  in  aircraft  construction.  The  third  study  produced 
a  forecast  of  laser  developments.  (B) 

GENERAL  TELEPHONE  AND  ELECTRONICS  CORPORATION 

Sponsor  of  the  Institute  for  the  Future  study:  Some  Prospects  for  So¬ 
cial  Change  by  1985  and  Their  Impact  on  Time/Money  Budget,  R-25,  (Enzer, 
Little,  and  Luzer) . 

GLAZIER,  FREDERICK,  P.(  et  al. 

"A  Multi-Industry  Experience  in  Forecasting  Optimum  Technological  Ap¬ 
proach" 

in  James  R.  Bright  and  M.E.F.  Schoeman  (eds.)  A  Guide  to  Practical  Tech¬ 
nological  Forecasting,  Prentice-Hall,  Inc.  Englewood  Cliffs,  New  Jersey, 
1973. 

This  paper  illustrates  how  the  author  used  Delphi  to  develop  a 
multi-industry,  multi-firm  approach  to  future  technological  requirements 
(Esso  and  Phillips  Petroleum).  Detailed  examples  of  questionnaires  and 
study  results  are  illustrated  in  the  paper.  (B) 

GOODMAN,  JOEL  M. 

"Delphi  and  the  Law  of  Diminishing  Returns" 

Technological  Forecasting  and  Social  Change  2,  1970,  p.  225. 
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The  purpose  of  this  brief  paper  la  to  outline  the  author's  thesis 
on  the  relationship  between  the  number  of  Delphi  studies  conducted  In 
an  area  and  the  amount  of  new  material  obtained  by  conducting  further 
studies.  In  this  brief,  he  mentions  that  Lockheed  Aircraft  Corp.  con¬ 
ducted  a  Delphi  study  "for  the  purpose  of  Identifying  e  olutlonary  and 
revolutionary  technological  concepts  that  might  be  anticipated  during 
the  remainder  of  the  twentieth  century."  No  results  of  the  study  are 
shown.  (B) 

GOODWILL,  DANIEL  Z. 

"A  Look  at  the  Future  Impact  of  Computer-Communications  on  Everyday 
Life" 

Delphi:  The  Bell  Canada  Experience,  Bell  Canada,  October  1972. 

The  computer  communication  revolution  will  have  an  impact  on  every 
aspect  of  our  lives.  Our  concepts  of  work,  work  locatio..,  home,  leisure 
will  undergo  some  rather  startling  changes  over  Hie  next  thirty  years. 
This  Is  the  view  of  a  group  of  individuals  who  participated  In  a  recent 
Bel]  Canada  Delphi  study.  It  is  very  important  for  leuders  In  all  sec¬ 
tors  of  the  economy  to  be  aware  of  the  implications  of  these  changes 
and  to  begin  thinking  about  how  we  can  implement  these  developments  In 
an  orderly  and  effective  manner. 

GOODWILL,  DANIEL  Z. 

An  Exploration  of  the  Future  Bueineee  Information  Processing  Technology 

Bell  Canada  Business  Planning,  October  1971.  (Proprietary) 

This  Delphi  first  explores  the  future  of  societal  values  and  ex¬ 
pected  changes  in  business  procedures.  Technological  services  examined 
include  management  information  services,  mini  and  small  computers,  data 
processing,  terminal  capabilities,  computer  and  coirmunicatlons  services 
In  the  home,  and  the  impact  of  technology  on  work  locations.  (B) 

GORDON,  T.  J. 

A  Study  of  Potential  Changes  in  Employee  Benefits ,  Volume  I:  Summary 
and  Conclusions 

Institute  for  the  Future,  R-l,  April  1969. 

GORDON,  T.  J. 

A  Study  of  Potential  Changes  in  Employee  Benefits,  Volume  II:  National 
and  International  Patterns 

Institute  for  the  Future,  R-2,  April  1969. 

GORDON,  T.  J. 

A  Study  of  Potential  Changes  in  Employee  Benefits,  Volume  III:  Delphi 
Study 

Institute  for  the  Future,  R-3,  April  1969. 

GORDON.  T.  J. 

A  Study  of  Potential  Changes  in  Employee  Benefits,  Volume  IV:  Appen¬ 
dices  to  the  Delphi  Study 

Institute  for  the  Future,  R-4,  April  1969,  cut  of  print. 
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GORDON,  T.  .1.,  and  R.  H.  AMENT 

Forecasts  of  Some  Technological  and  Scientific  Developments  and  Their 
Societal  Consequences 

Institute  for  the  Future,  R-6 ,  September  1969. 

GORDON,  T.  J.,  and  0.  HELMER 

Report  on  a  Long-Range  Forecasting  Study 

The  Rand  Corporation,  P-2982,  September  1964.  (Also  published  as  ap¬ 
pendix  to  0.  Helmer,  Social  Technology ,  Basic  Books,  New  York,  1966.) 

Description  of  an  experimental  trend-predicting  exercise  covering 
a  time  period  as  far  as  fifty  years  into  the  future.  The  Delphi  tech¬ 
nique  is  used  in  soliciting  the  opinions  of  experts  in  six  areas:  sci¬ 
entific  breakthroughs,  population  growth,  automation,  apace  progress, 
probability  and  prevention  of  war,  future  weapon  systems.  Possible  ob¬ 
jections  to  the  approach  are  aikO  discussed.  (R) 

GOULD,  JULIUS,  and  WILLIAM  L.  KOLB  (eds.) 

A  Dictionary  of  the  Social  Sciences 

Free  Press  of  Glencoe,  New  York,  1964. 

GRABBE,  E.  M. ,  and  D.  L.  PYKE 

"An  Evaluation  of  the  Forecasting  of  Information  Processing  Technology 
and  Applications" 

Journal  of  Technological  Forecasting  and  Social  Change ,  (to  be  published) 
HALL,  P.  D. 

"Technological  Forecasting  for  Computer  Systems"  In  G.  S.  C.  Wills 
et  al.,  (eds.),  Technological  Forecasting  and  Corporate  Strategy, 

Bradford  University  Press,  1969. 

Discusses  technological  forecasting  and  Delphi.  Several  charts 
shoving  the  computer  Delphi  for  ICL  are  shown  as  well.  (B) 

HALL,  T.  W. 

"Implementation  of  an  Interactive  Conference  System" 

AFIPS  Conference  Proceedings ,  Vol.  38,  1971  Spring  Joint  Computer  Con¬ 
ference  held  from  May  18-20,  1971  in  Atlantic  City,  New  Jersey,  Mont- 
vale,  New  Jersey,  AFIPS  Press,  1971,  pp.  217-229. 

HAYDEN,  SPENCER 

"Hov  Industry  Is  Using  Technological  Forecasting" 

Management  Review,  May  1970,  pp.  4-15. 

This  article  reports  on  the  author's  survey  of  65  "progressive" 

(no  definition)  companies  and  their  experiences  with  technological  fore¬ 
casting.  Thirty  methods  were  cited  by  the  companies.  While  panel  con¬ 
sensus  methods  (consulting  groups  of  experts  in  person  or  writing)  were 
found  in  69  percent  of  the  companies,  formal  Delphi  was  rated  relatively 
unknown.  Heyden  found  that  26  percent  of  the  firms  had  used  Delph*  and 
that  71  percent  of  these  firms  had  proved  it  useful.  There  are  not  any 
specific  company  references  or  sample  Delphi  results  in  the  paper.  (B) 

HAYDON,  B.  V. 

The  Year  2000 

The  Rand  Corporation,  P-3571,  March  1967. 
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A  survey  of  the  possible  future  of  the  year  2000  and  the  problems 
that  advancing  technology  pose  for  civilized  man.  Rand's  Delphi  tech¬ 
nique  and  the  implications  of  some  of  the  predictions  obtained  by  the 
method  are  discussed:  over-population  and  food  production;  nuclear 
power  as  a  source  of  energy;  air  pollution;  weather  control;  automation, 
education,  and  the  home.  Two  compelling  reasons  exist  for  looking  into 
the  future:  to  detect  danger  signals  so  that  action  can  be  taken  to 
forestall  unpleasant  events  or  conditions,  and  to  avoid  making  mistakes. 
Decisions  made  today  directly  affect  the  future,  and  rational  man  must 
view  the  future  as  subject  to  his  control.  (R) 

-HELLER,  F.  A. 

"Croup  Feedback  Analysis:  A  Method  of  Field  Research" 
iajchological  Bulletin,  Vol.  72,  No.  2,  1969,  pp.  108-117. 

"Group  feedback  analysis"  combines  a  number  of  features  from  nomo¬ 
thetic  and  idlographlc  methods,  thus  overcoming  some  of  the  limitations 
of  the  mailed  questionnaire.  Certain  advantages  were  described  by  ref¬ 
erence  to  a  number  of  research  problems.  Group  feedback  analysis  has 
three  interrelated  stages:  (1)  individual  research  instruments  are  ad¬ 
ministered  to  a  group  of  subjects;  (2)  some,  or  all,  of  the  results  from 
Stage  1  are  fed  back  to  the  group  as  means  and  deviations;  and  (3)  a 
discussion  based  on  feedback  of  results  is  stimulated,  recorded,  and 
later  content  analyzed.  Moving  from  unstructured  to  focused  question¬ 
ing,  the  investigator  obtains  a  varied  amount  of  information  which  acts 
both  as  a  check  and  extension  of  the  results  obtained  from  Stage  1. 

HELMER,  0.,  and  H.  HELMER 

Future  Opportunities  for  Foundation  Support 

Institute  for  the  Future,  R-ll,  June  1970. 

HELMER,  0. 

I'Ontj-Banye  Forecasting — Roles  and  Methods 

Institute  for  the  Future,  P-7,  May  1970. 

HELMER,  0. 

Systematic  Use  of  Expert  Opinions 

The  Rand  Corporation,  P-3821,  November  1967. 

An  explanation  of  the  experimental  Delphi  technique,  a  systematic 
procedure  for  obtaining  the  opinions  of  experts  on  a  particular  subject. 
Four  sets  of  questionnaires  are  used,  each  asking  for  successive  refine¬ 
ments  in  the  estimated  answer  to  a  given  question.  The  interquartile 
range  of  the  response  is  crucial.  Respondents  outside  this  range  are 
invited  to  defend  or  reevaluate  their  answers,  using  the  information 
feedback  available.  Refinements  include  subsidiary  questions,  attribu¬ 
tion  of  differential  weights  to  opinions,  and  the  removal  of  systematic 
bias.  Future  applications  may  use  automatic  processing  for  opinions  of 
panel  members  geographically  remote  from  each  other.  Simplified  ver¬ 
sions  of  the  Delphi  technique  can  be  used  in  face-to-face  discussion; 
more  complex  versions  tap  the  panelists'  intuitive  knowledge  through 
bo t rarchical  sets  of  expert  opinions.  (R) 
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HELMER.  0. 

Prospects  of  Technological  Progress 

The  Rend  Corporation,  P-3643,  1967. 

Discusses  the  role  that  the  forecasting  of  technological  develop¬ 
ments  plays  in  shaping  the  future  of  our  society.  Gives  some  predic¬ 
tions  for  the  year  2000.  Forecasting  is  an  aid  to  decisionmakers,  and 
can  effect  the  future.  Analysis  of  the  future  requires:  (1)  a  survey 
of  alternatives;  (2)  an  analysis  of  preferences;  (3)  constructive  policy 
research.  He  feels  the  prospects  of  socio-technological  progress  dur¬ 
ing  the  next  third  of  this  century  are  very  high,  due  to  increased  sci¬ 
entific  manpower,  the  computer  and  a  reorientation  toward  policy-related 
research.  Advent  of  social  technology.  Maybe  a  comprehensive  theory 
of  organizations. 

"...  we  may  well  look  forward  to  the  emergence  cf  a  new  breed  of 
modern-day  constructive  Utopians."  (P) 

HELMER,  0. 

Methodology  of  Societal  Studies 

The  Rand  Corporation,  P-3611,  June  1967. 

The  qualitative  improvement  of  societal  studies  depends  upon  all- 
out  acceptance  of  operations  research  techniques,  adoption  of  a  systems 
approach  as  a  basic  principle,  a  real  effort  toward  interdisciplinary 
collaboration,  and  deliberate,  intensive  orientation  toward  the  future. 
Competency  in  the  techniques  associated  with  the  new  methodology  will 
be  required.  (R) 

HELMER,  0. 

The  Future  of  Science 

The  Rand  Corporation,  P-3607,  May  1967. 

The  application  of  the  techniques  of  the  physical  sciences  to  the 
social  sciences,  coupled  with  the  increasing  capability  and  refinement 
of  the  computer,  is  presented  as  a  possible  method  of  solving  socio¬ 
political  problems  in  the  future.  The  increasingly  symbiotic  relation¬ 
ship  between  man  and  machine  and  the  refinement  of  interdisciplinary, 
operations-analytical  techniques  in  Che  social  sciences,  particularly 
the  Delphi  technique  and  game  theory  will  produce  breakthroughs  and 
new  forms  ot  procedure  in  the  scientific  establishment.  (R) 

HELMER,  0. 

A’eu  Developments  in  Early  Forecasting  of  Public  Problems:  A  New  Intel¬ 
lectual  Climate 

The  Rand  Corporation,  P-3576,  April  1967.  (Also  in  Vital  Speeches , 

Vol.  33,  1967,  pp.  497-499.) 

A  report  of  philosophical,  pragmatic,  and  methodological  changes 
in  world  attitude  toward  the  future — all  favoring  positive  long-range 
planning.  The  second  computer  revolution  is  leading  to  true  man-machine 
symbiosis.  The  aociel  sciences  are  turning  to  an  interdisciplinary 
systems  approach  to  the  solution  of  socio-political  problems,  using 
astheastlcsl  models,  simulation  procedures,  and  a  systematic  approach 
to  the  utilization  of  expert  opinions.  To  arrive  at  a  positive  payoff 
for  all  requires  reasonable  expectation  that  cooperative  moves  will 
meet  with  a  cooperative  response.  The  revolutionary  reorientation  in 
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the  social  acioncee  gives  hope  that  in  the  next  generation  this  attitude 
will  extend  into  the  uncivilised  area  of  international  relatione.  (Pre¬ 
pared  for  preaentation  at  a  Public  Affairs  Perspective  Conference  under 
National  Industrial  Conference  Board  auspices.  New  York,  April  1967.) 

00 

H ELMER,  0. 

Analy8ia  of  the  Future:  The  Delphi  Method 

The  Rand  Corporation,  P-3558,  March  1967. 

A  description  of  the  Delphi  technique  which  attempts  to  make  ef¬ 
fective  use  of  informed  intuitive  judgment  in  long-range  forecasting. 

The  Delphi  method  in  its  simplest  form  solicits  the  opinions  of  experts 
through  a  series  of  carefully  designed  questionnaires  Interspersed  with 
information  and  opinion  feedback.  A  convergence  of  opinion  has  been 
ohserved  in  the  majority  of  cases  where  the  Delphi  approach  has  been 
used.  In  a  few  of  the  cases  where  no  convergence  toward  a  relatively 
narrow  Interval  of  values  took  place,  opinions  began  to  polarize  around 
two  distinct  values;  two  schools  of  thought  regarding  a  particular  is¬ 
sue  seamed  to  emerge.  Refinements  that  have  been  made  in  the  Delphi 
technique  consist  of  the  introduction  of  weighted  opinions  and  use  of 
the  technique  in  conjunction  with  a  simulated  decisionmaking  process. 

(R) 

HELMER,  0. 

A  Uae  of  Simulation  for  the  Study  of  Future  Values 

The  Rand  Corporation,  P-3443,  September  1966. 

HELMER,  0. 

The  Uee  of  the  Delphi  Technique  in  Problems  of  Educational  Innovations 

The  Rand  Corporation,  P-3499,  December  1966. 

A  description  of  the  Delphi  technique,  a  method  for  the  systematic 
solicitation  and  collation  of  expert  opinions,  and  its  applications  to 
educational  planning.  Delphi  pilot  experiments  carried  out  in  an  Edu¬ 
cational  Innovations  Seminar,  UCLA,  apply  the  Delphi  technique  of  long- 
range  forecasting  to  proposals  for  innovations  in  educational  methods 
and  budget  allocations  to  achieve  these  innovations.  (R) 

HELMER,  0. 

Social  Technology 

Basic  Books,  New  York,  1966.  (See  also  The  Rand  Corporation,  P-3063, 
February  1965. 

A  reappraisal  of  methodology  in  the  social  sciences  with  specific 
proposals  for  awdificatione  of  traditional  procedures.  The  paper  sug¬ 
gests  that  social  scientists  explore  the  possibilities  of  operations 
research  approaches,  of  operational  model  building,  and  of  expert  opin¬ 
ion.  It  also  suggests  procedures  in  the  areas  of  urban  renewal,  educa¬ 
tional  reform,  political,  and  long-range  economic  forecasting.  (R) 

HELMER,  0. 

Convergence  of  Expert  Consensus  Through  Feedback 

The  Rand  Corporation,  P-2973,  September  1964. 
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A  discussion  of  studies  directed  toward  the  Improved  use  of  expert 
opinions  in  operations  research.  The  Delphi  method  as  applied  to  con- 
sansus  research  is  discussed  and  the  results  of  experiments  directed 
toward  convergence  of  expert  opinions  presented.  (R) 

HELMEP.,  0. 

The  Systematic  Uee  of  Expert  Judgment  in  Operations  Research 

The  Rand  Corporation,  P-2795,  September  1963. 

A  consideration  of  operations  research  as  a  science,  though  an  In¬ 
exact  one.  The  operations  analyst,  as  opposed  to  the  pure  scientist, 
emphasises  control  rather  than  understanding.  Judgment  In  constructing 
and  applying  operatlous-analytl :al  models  should  be  as  expert  and  its 
applications  as  systematic  as  possible.  Improved  methods  of  identifying 
and  measuring  expertness  and  of  employing  experts  efficiently  are  needed. 
In  particular,  the  use  of  groups  of  experts  by  consensus  techniques,  by 
the  Delphi  technique,  and  by  simulation  procedures  and  operational  gam¬ 
ing  should  be  further  refined.  (R) 

HELMER,  0. 

The  Outcome  of  a  Recent  Experiment  in  Prediction 

The  Rand  Corporation,  1952.  (Unpublished) 

HELMER,  0,  T.  J.  GORDON,  S.  ENZER,  R.  DE  BRIGARD,  and  R.  ROCHBF.RG 
Development  of  Long-Range  Forecasting  Methods  for  Connecticut ;  A  Sum¬ 
mary 

Institute  for  the  Future,  R-5 ,  September  1969. 

HELMER,  0.,  and  E.  S.  QUADE 

An  Approach  to  the  Study  of  a  Developing  Economy  by  Operational  Gaming 

The  Rand  Corporation,  P-2718,  1963. 

Thla  la  one  of  the  earlier  papers.  It  "...  considers  the  possible 
use  of  operatlonsl  gaming,  or  simulation  using  human  players,  to  exam¬ 
ine  an  aconomy  as  s  whole."  "(Operational  gaming]  undoubtedly  Is  most 
useful  when  applied  with  a  clear  objective  in  mind  to  well-structured 
problems  based  on  abundant  data."  However,  there  is  more  chance  in  a 
developing  economy  of  applying  operations  research  to  "strategic"  prob¬ 
lems.  The  model  is  seen  essentially  as  a  communications  medium.  Pro¬ 
vides  a  good  argument  for  use  of  simulation  before  one  has  an  adequate 
theory.  Diecuaees  computer  simulation,  and  the  objections  to  it,  then 
advantages  of  gaming,  where  use  of  computers  is  minimal,  reliance  on 
the  intuitive  expertise  of  specialists,  and  emphasis  is  on  clearer 
problem  formulation.  For  gaming,  break  down  economy  into  sectors. 

Include  intangibles,  also  gat  feedback. 

Thay  do  not  see  any  iissadlately  usable  output.  (P) 

HELMER,  0.,  and  N.  H.  RES CHER 

On  the  Epistemology  of  the  Inexact  Sciences 

The  Rand  Corporation,  R-353,  February  1960. 

A  nav  epistemological  approach  to  the  inexact  sciences,  which  in¬ 
clude  applied  phyalcal  aclencaa,  auch  aa  engineering  and  medicine,  as 
well  as  moat  of  the  social  science*.  The  purpose  of  all  science  is  to 
explain  past  events  and  to  predict  future  ones  in  an  objective  manner. 
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While  explanation  and  prediction  have  the  sane  logical  structure  in  the 
exact  sciences i  this  is  not  so  in  the  inexact  sciences.  This  fact  leads 
to  the  development  of  specifically  predictive  instrumentalities  in  these 
fields  and  to  various  methodological  innovations.  Among  these  ere  the 
systematic  employment  of  expert  Judgment  and  the  use  of  pseudr^xperi- 
mentation,  Involving  simulation  processes ,  and  in  particular,  opera¬ 
tional  gaming.  (R) 

HERCULES  POWER  COMPANY 

See  references  by  PARKER  and  WILLS  for  elaboration. 

HERSCH,  CHARLES 

"The  Discontent  Explosion  in  Menial  Health" 

American  Psychologist,  1969,  pp.  597-606. 

HILLS,  L.  S. 

"Delphi  Technique:  A  Tool  for  Business  Forecasting" 

A  ACE  Bulletin,  Vol.  12,  No.  2,  1970,  p.  48. 

HONEYWELL 

See  the  Alderaon  and  Sproull  article  describing  the  use  of  the  PAT¬ 
TERN  Technique  at  Honeywell.  This  technique  1b  partly  related  to  Delphi 
as  it  uses  panels  of  experts  to  develop  and  relevance  numbers.  The  Me- 
Clauchlin  paper  describes  some  Honeywell  Delphi  experience  in  more 
detail.  (B) 

I CL  (INTERNATIONAL  COMPUTERS  LTD.) 

See  references  and  results  summarized  by  WILLS  and  by  HALL. 

IFIP- INTERNATIONAL  FEDERATION  OF  INFORMATION  PROCESSING 

See  the  study  conducted  by  PARSONS  and  WILLIAMS-— 'Forecast  1968- 
2000  ..." 

IKLE ,  F.  C. 

Social  Forecasting  and  the  Problem  of  Changing  Values,  tilth  Special 
Hcfnrence  to  Soviet  and  East  Euj'opcan  Writings 

The  Rand  Corporation,  P-4550,  January  1971. 

JANTSCH,  E. 

'Technological  Forecasting  in  Perspective :  A  Frameuork  for  Technological 
Forecasting,  Its  Techniques,  and  Organisation:  A  Description  of  Activ¬ 
ities  and  Annotated  Bibliography 

Organization  for  Economic  Cooperation  and  Development ,  Paris,  1967. 

THE  JOHNS  HOPKINS  UNIVERSITY 

"Provide  Prognostic  Epidemiologic  Estimates" 

Department  of  Medical  Care  and  Hospitals,  1970. 

J0LS0N,  MARVIN  A.,  and  GERALD  L.  R0SS0W 

"The  Delphi  Process  in  Marketing  Decision  Making" 

of  Marketing  Research  Vol.  8,  November  1971,  pp.  443-448. 
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Describes  the  use  of  Delphi  In  an  experiment  conducted  for  the  Pace 
Computing  Corporation  in  estimating  market  demand  for  its  services. 

Part  of  the  article  describes  the  authors'  evaluation  of  the  techniques 
(similar  to  Rand  experiments).  Concludes  that  the  technique  can  be  U9ed 
for  marketing  purposes  and  as  an  internal  communications  tool  in  a  bus¬ 
iness.  (B) 

KAPLAN,  A.,  A.  L.  SKOCSTAD,  and  M.  A.  GIRSHICK 
"The  Prediction  of  Social  and  Technological  Events" 

Public  Opinion  Quarterly ,  Vol.  14,  No.  1,  1930,  pp.  93-110. 

(See  also  The  Rand  Corporation,  P-93,  1949.) 

A  group  of  individuals  with  a  generally  high  education  level  was 
asked  to  make  predictions  concerning  a  large  number  of  future  events  in 
order  to  investigate  certain  aspects  of  the  use  of  expert  opinion  in 
policymaking.  In  particular,  the  following  questions  were  explored: 
how  good  are  expert  predictions  in  areas  germane  to  policy;  how  can 
such  predictions  be  improved;  and  how  can  the  reliability  of  a  given 
opinion  be  appraised  beforehand?  While  the  design  of  the  st  dy  makes 
projection  of  its  results  on  other  situations  questionable,  it  throws 
considerable  light  on  the  problems  involved.  Among  other  results,  it 
was  found  that  confidence  in  prediction  does  not  necessarily  show  a 
correlation  with  success  in  prediction,  that  predictions  tnaae  by  groups 
of  people  are  more  likely  to  be  right  than  predictions  made  by  the  same 
individuals  working  alone,  and  that  the  reliability  of  predictions  can 
be  appraised  to  some  extent  by  examining  the  character  of  the  justifi¬ 
cations  given  for  them. 

The  success  of  the  collective  psychological  methods,  i.e.,  group 
discussion  and  iteration,  was  duplicated  by  using  statistical  methods 
such  as  averaging  on  the  individual  results.  (P) 

KIMBLE,  R, 

Delphic  Forecasting  Critical  Personnel  Requirement.-. 

II. S.  Amy  Electronics  Command,  Fort  Monmout1-,  N*w  Jersey,  November  1969. 

KNOCK,  RICHART  T. 

Electro -Optical  Technology 

Long  Range  Planning  Service  Report  468,  Stanford  Research  Institute, 
August  1972,  (Client  proprietary). 

This  report  illustrates  how  Delphi  material  can  be  Integrated  with 
other  data  in  a  research  report  on  a  specific  subject.  The  Delphi  data 
in  this  report  was  obtained  from  the  TRW  PROBE  study  and  from  the  Jap¬ 
anese  Science  and  Technology  Agency — "Report  on  Technological  Goods; 
September  1971."  The  Delphi  summary  chart  presents  43  forecasts  on  13 
major  industrial  applications  of  electro-optical  technology  (pages  8- 
9).  (B) 

KOCHMAN,  A.  F. 

An  Investigation  of  the  Delphi  Forecasting  Technique  with  Emphasis  Upon 
Educational  Need  Assessment 

Master's  Thesis,  Chapman  College,  1968. 
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KOPKIND,  A. 

"The  Futuie-Planners" 

The  New  Republic,  February  1967,  pp.  19-23. 

LACHMANN,  OLE 

"Personnel  Administration  in  1980:  A  Delphi  Study" 

Long-Range  Planning ,  Vol.  5,  No.  2,  Oxford,  England,  June  1972. 

This  article  reviewn  two  studies  on  trends  in  personnel  management. 
This  Danish  study  was  first  prepared  as  Input  to  a  conference.  Sample 
results  are  shown.  The  second  study  was  an  extension  of  the  first  one 
but  using  employees  from  the  printing  firm  CON-FORM.  Results  from  the 
CON-FORM  study  are  also  shown  in  the  paper.  (B) 

LANSDOWNE,  Z.  F. 

Analyeie  of  Intercity  Transport  Improvement  a :  Forecasting  Demand  and 
Evaluation  User  Benefits , 

The  Rand  Corporation,  RM-6255,  May  1970. 

LIKERT,  R., 

The  H’uman  Organization 

Me Craw  Hill,  New  York,  1967. 

LING-TEMCO-VAUGHT  INC.  (LTV) 

Reference  and  sample  questionnaire  page  shown  in  Wills  Technologi¬ 
cal  Forecasting.  Martino  also  references  this  study  in  his  book.  The 
Bright  book  "A  Brief  Introduction..."  usas  the  LTV  Delphi  extensively 
as  a  case  example.  Detailed  forecasts  and  methodology  is  outlined. 
Cetron  also  uses  LTV  experience  in  his  book.  (B) 

LING-TEMCO-VOUGHT 
Continuing  Use  of  the  Delphi 

Proprietary  document. 

LORGE,  I  M.,  D.  FOX,  J.  DAVITZ,  and  T.  BRENNER 

"Survey  of  Studies  Contrasting  the  Quality  of  Group  Performance  and  In¬ 
dividual  Performance,  1920-1957" 

Psychological  Bulletin,  Vol.  55,  1958,  pp.  337-372. 

Research  contrasting  the  quality  of  group  performance  with  indi¬ 
vidual  performance  in  each  of  the  following  general  to, ic  areas  has 
been  examined  in  this  paper:  judgment,  learning,  social  facilitation, 
problem  solving,  memory,  size  of  group,  problem  solving  in  more  realis¬ 
tic  situations,  and  productivity.  Recent  theoretical  and  methodological 
considerations  as  well  as  discussions  of  group  types  are  included.  Re¬ 
search  weaknesses  and  theoretical  problems  are  discussed. 

LOCKHEED  AIRCRAFT  CORPORATION 

See  reference  in  paper  by  GOODMAN. 

LUDLOW,  J.  D. 

Geo.  Grant  Delphi  Exercises:  Techniques  for  Utilizing  Informed  Judgments 
tjf  a  Multi-Disciplinary  Team  of  Researchers 

Bureau  of  Business  Research,  University  of  Michigan,  Working  Paper  22, 
January  1971. 
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Application  of  Delphi  to  regional  and  urban  planning. 

MAQilLLAN  BLOEDEL  LTD. 

Sponsor  of  the  Inatitute  for  the  Future  Study  R-16:  The  Future  of 
Newsprint  (Paul  Baran). 

MAIER,  N.  R.  F. 

"Assets  and  Liabilities  in  Croup  Problem  Solving:  The  Need  for  an  In¬ 
tegrative  Function" 

Psychological  Review,  Vol.  74,  No.  4,  July  1967,  pp.  239-249. 

Research  on  group  problem  solving  reveals  that  the  group  has  both 
advantages  and  disadvantages  over  Individual  problem  solving.  If  the 
potentials  for  group  problem  solving  can  be  exploited  and  if  its  defi¬ 
ciencies  can  be  avoided,  it  follows  that  group  problem  solving  can  at¬ 
tain  a  level  of  proficiency  not  ordinarily  achieved.  The  requirement 
for  achieving  this  level  of  group  performance  seems  to  hinge  on  develop¬ 
ing  a  style  of  discussion  leadership  which  maximizes  the  group's  assets 
and  minimizes  its  liabilities.  Since  members  possess  the  essential  in¬ 
gredients  for  the  solutions,  the  deficiencies  that  appear  in  group  so¬ 
lutions  reside  in  the  processes  by  which  group  solutions  develop.  These 
processes  can  determine  whether  the  group  functions  effectively  or  in¬ 
effectively.  The  critical  factor  in  a  group's  potential  is  organization 
and  integration.  With  training,  a  leader  can  supply  these  functions 
and  serve  as  the  group's  central  narvoua  system,  thus  permitting  the 
group  to  emerge  as  a  highly  efficient  entity.  Extensive  bibliography. 
(P) 

MARIEN,  M.  (Ed.) 

The  Hot  List  Delphi:  An  Exploratory  Survey  of  Essential  Reading  for 
the  Future 

Educational  Policy  Research  Center,  Syracuse  University  Research  Cor¬ 
poration,  1972. 

A  report  Hating  236  books  and  articles,  of  which  192  were  rated 
by  a  panel  of  fourteen  futurists.  "Recommended  to  anyone  seeking  guid¬ 
ance  to  the  best  books  to  read  in  futurlstlca." 

MARTINO,  J.  P. 

"How  the  Soviets  Forecast  Technology" 

The  Futurist ,  Vol.  7,  No.  1,  February  1973,  pp.  30-31. 

MARTINO,  J.  P. 

Technological  Forecasting  for  Decisionmaking 

American  Elsevier,  New  York,  1972. 

This  detailed  Treatment  of  the  subject  (750  pp.)  examines  all  as¬ 
pects  of  technologic* i  io».Hc«ntfng.  Tha  chapter  on  Delphi  uses  corpor¬ 
ate  studies  as  examples  in  outlining  how  Delphi  studies  have  been  con¬ 
ducted.  Businese  studies  shown  include  those  of  Sulth,  Kline,  and 
French,  TRW,  Parsons  and  Will la**,  and  LTV.  Many  of  the  specific  an¬ 
alyses  of  the  technique  Itself  sere  based  upon  examining  the  raw  TRW 
date  provided  to  Martino..  (B) 
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MARTINO,  J.  P. 

"Technological  Forecasting  is  Alive  and  Well  in  Industry" 

The  Futurist ,  August  1972. 

In  this  article  Martino  discusses  the  activities  of  several  com¬ 
panies  who  have  maintained  technological  forecasting  activities  during 
the  recent  economic  recession  while  other  firms  stopped  these  studies. 
Delphi  studies  at  the  Convair  Division  of  General  Dynamics  are  outlined. 
The  Bell  Canada  use  of  Delphi  study  results  for  Business  Planning  pur¬ 
poses  is  also  discussed.  Martino  concludes  that  the  survival  of  tech¬ 
nological  forecasting  in  the  corporate  environment  is  dependent  upon 
making  the  results  useful  for  decisionmaking  purposes.  (B) 

MARTINO,  J.  P. 

"What  Computers  May  Do  Tomorrow" 

The  Futurist ,  October  1969,  p.  134-135. 

The  author  reviews  the  Parsons  and  Williams  Delphi  study.  The 
first  part  of  the  article  outlines  the  background  and  conduct  of  the 
study  (panel  size,  number  of  rounds  and  events  forecast,  panel  make-up, 
etc.).  The  second  part  reviews  some  of  the  study  findings.  One  chart 
is  Included.  (B) 

MARTINO,  J.  P. 

An  Experiment  with  the  Delphi  Procedure  for  Long-Range  Forecasting 

Office  of  Scientific  Research,  U.S.  Air  Force,  AFSOR  670175,  1967. 

MAS LOW,  ABRAHAM 

The  Farther  Reaches  of  Human  Nature:  Towards  A  Psychology  of  Being 

Viking  Press,  New  York,  1971,  Princeton,  N.  J.,  Van  Nostrand,  1962, 

1968,  Second  Edition. 

MASON,  R.  0. 

"A  Dialectical  Approach  to  Strategic  Planning" 

Management  Science,  Vol.  15,  No.  8,  pp.  B-403-414,  1969. 

MCDONNEL  DOUGLAS,  DOUGLAS  AIRCRAFT  DIVISION 

Referenced  in  the  Business  Week  article.  This  study  forecast  the 
future  of  commercial  air  transportation.  Revenue  forecast  on  passenger 
and  cargo  operations  were  developed.  One  purpose  of  their  forecasts 
is  to  determine  the  date  when  all-cargo  jumbo  jets  should  be  Introduced. 
(B) 

MCGLAUCHLIN,  LAURENCE  D. 

"Technological  Audits:  An  Aid  to  Research  Planning"  in  James  R.  Bright 
and  M.  E.  F.  Schoeman  (eds.),  A  Guide  to  Practical  Technological  Fore¬ 
casting,  Prentice-Hall,  Inc.,  Englewood  Cliffs,  New  Jersey,  1973. 

A  use  of  Delphi  at  Honeywell  for  corporate-wide  technological  fore¬ 
casting  efforts  Is  described.  This  use  of  Delphi  is  part  of  an  overall 
process  of  R&D  planning  at  Honeywell.  Questionnaire  samples,  the  use 
and  value  of  Delphi  as  an  internal  communications  tool  is  also  outlined. 
(B) 
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MCGREGOR,  DOUGLAS 

"The  Major  Determinants  of  the  Prediction  of  Social  Events" 

Journal  of  Abnormal  aid  Social  Psychology,  Vol.  33,  1938,  pp.  179-204. 

MCLOUGHLIN,  W.  G. 

Product  Cycle  Planning 

Presented  at  the  Technological  Forecasting  Conference,  Lake  Placid, 

New  York,  1969. 

MILKOVICH,  C.  T.,  A.  J.  ANNONI,  and  T.  A.  MAHONEY 
The  Uee  of  the  Delphi  Procedures  in  Manpower  Forecasting 
Minnesota  University  Minneapolis  Center  for  the  Study  of  Organizational 
Performance  and  Human  Effectiveness,  TR-7007  (AD747  651),  1972.  Pre¬ 
sented  at  American  Meetings  of  the  Institute  of  Management  Sciences, 
October  12,  1971. 

A  case  study  was  made  in  the  development,  implementation  and  eval¬ 
uation  of  the  Delphi  technique,  which  systematically  makes  use  of  ex¬ 
pert  judgment  in  generating  manpower  forecasts.  This  case  study  was 
conducted  in  a  large  national  retail  organization  on  professional  man¬ 
power.  The  results  of  the  Delphi  technique  using  an  expert  and  a  naive 
panel  are  compared  with  results  generated  by  a  conventional  regression 
based  model  and  the  actual  experience  of  the  organization,  which  serves 
as  the  criterion.  The  study  also  analyzes  the  informational  elements 
used  by  experts  during  the  Delphi  procedures  and  develops  a  model  based 
on  these  elements.  The  usefulness  of  the  Delphi  in  generating  manpower 
forecasting  models  is  discussed. 

MITROFF,  I.  I. 

"A  Communication  Model  of  Dialectical  Inquiring  Systems — A  Strategy  for 
Strategic  Planning" 

Management  Sciencee,  Vol.  17,  No,  10,  June  1971,  pp.  B-6 34-648. 

MONSANTO 

CETRON  references  the  Monsanto  work. 

MOORE,  C.  G.,  and  H.  P.  POMREHN 

Technical  Foreaaeting  of  Marine  Transportation  Systems  1970-2000 

University  of  Southern  California,  June  1969. 

MORRIS,  P.  A. 

Bayesian  Expert  Resolution 

Doctoral  Dissertation,  Stanford  University,  1971. 

NANUS,  BURT,  LELAND  M.  WOOTEN,  and  HAROLD  B0RK0 

The  Social  Implications  of  the  Use  of  Computers  Across  National  Bound¬ 
aries 

AFIPS  Press,  Montvale,  Mew  Jersey,  1973  (forthcoming). 

In  all,  more  than  seventy  areas  of  potential  impact  were  explored 
with  the  panel.  The  panel  strongly  agreed  that  world  economic  develop¬ 
ment  will  be  greatly  affected  by  the  globalization  of  information  proc¬ 
essing  systems  and  that  the  use  of  computers  across  national  boundaries 
in  both  the  public  and  the  private  sectors  will  expand  very  greatly  in 
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the  next  two  dacadea.  Among  the  many  specific  conclusions  of  the  panel 
are  the  following: 

1.  New  institutions  will  be  required  at  the  multinational  level 
to  resolve  disputes  over  the  transmission  of  data  across  national  bound¬ 
aries,  to  develop  regulations  concerning  the  activities  of  multinational 
data  banks,  to  provide  individual  safeguards,  and  to  deal  with  the  prob¬ 
lems  of  standardisation  of  data  tranaaiasion  facilities  and  capabilities. 

2.  The  use  of  multinational  computer  systems  will  tend  to  enhance 
the  power  of  multinational  corporations  vis-a-vis  the  nation-state  while 
at  the  saae  tine  contributing  to  a  growing  uniformity  of  business  prac¬ 
tices  throughout  the  world. 

3.  Within  the  highly  industrialized  societies,  many  people  will 
find  themselves  in  some  form  of  man-machine  relationship,  often  involv¬ 
ing  aultlnatlonal  communications,  within  the  next  decade.  These  inter¬ 
actions  may  be  for  education,  health,  library,  business  or  other  reasons 
but  the  net  effects  will  be  the  enhanceaent  of  shared  beliefs  and  val¬ 
ues  and  a  growing  sense  of  interdependence  on  matters  of  the  most  fun¬ 
damental  nature. 

4.  The  use  of  multinational  computer  systems  may  tend  to  enhance 
the  economic  interests  of  the  information  rich,  wealthier  nations  at 
the  expense  of  the  information  poor,  but  in  the  long  run,  the  use  of 
such  systems  will  increase  the  technological  options  available  to  the 
leas  daveloped  countries  and  apeed  up  their  ability  to  industrialize. 

NATIONAL  RESEARCH  COUNCIL,  NATIONAL  MATERIALS  ADVISORY  BOARD,  COMMITTEE 
ON  TECHNOLOGICAL  PORECASTING 

Technological  Forecasting  and  Engineering  Materials 

NMAB-279,  December  1970. 

Comparison  of  Delphi  with  other  planning  tools. 

NATIONAL  TRAINING  LABORATORIES 

Croup  Development 

(Bradford,  Leland  Powers),  Washington  National  Training  Laboratory, 
National  Education  Association,  1961. 

NORTH,  H.  Q. 

A  Probe  of  TRV'e  Future:  The  Next  20  Years 

TRW  Systems,  July  1969,  (proprietary  document). 

NORTH,  H.  Q. 

Delphi  Forecasting  in  Electronics 

Thompson  Remo ,  Wooldridge,  1968,  unpublished. 

NORTH,  H.  Q.,  and  D.  L,  Pyke 
"Probes  of  the  Technological  Future" 

Harvard  Business  Review,  Vol.  47,  May  1969,  pp.  68-82. 

Implications  of  technological  change.  Step-through  of  the  PROBE 
and  PROBE  2  Delphi  studies.  Similar  charts  to  IEEE  article.  Appendix 
on  the  basic  Delphi  method.  (B) 

NORTH,  H.  Q.,  and  D.  L.  PYKE 

"Technological  Forecasting  in  Planning  for  Company  Growth" 

IEEE  Spectrum,  January  1969. 
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A  review  of  the  uee  of  TRW's  Delphi  modification  (PROBE).  The 
planning  eye tea  at  TRW  is  alao  Illustrated  in  several  detailed  flow 
charts.  No  detailed  results  of  the  studies  are  ahown.  (B) 

NORTH,  H.  Q.,  and  D.  L.  PYKE 

"Technology,  the  Chicken — Corporate  Goals,  the  Egg" 

Technological  Forecasting  for  Industry  and  Government ,  Jaaes  R.  Bright 
(ed.),  Prentice-Hall,  Englewood  Cliffs,  New  Jersey,  1968. 

Technology  and  gcsls  are  interrelated,  rather  than  one  causing  rhe 
other.  (P) 

OFFICE  OF  HEALTH  ECONOMICS 

Medicines  in  the  1990's — A  Technological  Forecast 

London,  October  1969. 

OVERBURY,  R.  E. 

"Technological  Forecasting:  A  Criticise  of  the  Delphi  Technique" 
Long-Range  Planning,  Vol.  1,  No.  4,  June  1969,  pp.  76-77. 

OWENS-CORNING  FI BERG LAS  CORPORATION 

Sponsor  of  two  Institute  for  the  Future  studies:  R-13,  Some  Pros¬ 
pects  for  Residential  Housing  by  1986  (ENZER)  and  R-17,  Some  Develop¬ 
ments  in  Plastics  and  Competing  Materials  by  1936  (ENZER). 

PACE  COMPUTING  CORPORATION 

See  article  by  JOLSON  and  ROSSOW. 

PACKARD,  KARLE  S. 

” lapse t  of  an  Emerging  Technology  on  Company  Operations" 

A  Guide  to  Practical  Technological  Forecasting,  Janes  R.  Bright,  and 
M.  E.  F.  Schoenan,  (eds.),  Prentice-Hall,  Inc.  Englewood  Cliffs,  New 
Jersey,  1973. 

The  author  describes  the  use  of  a  Delphi  study  as  part  of  a  tech¬ 
nological  forecasting  exercise  at  AIL.  The  forecasting  exercise  was 
concerned  with  the  nuaber  of  electronic  circuits  that  could  be  Included 
on  LSI  chips.  Other  techniques  used  were  horizontal  decision  matrices, 
trend  extrapolations,  and  substitution  analysis.  (Pi 

PARKER,  E.  F. 

"British  Chemical  Industry  In  the  1980' s— A  Delphi  Method  Profile" 
Chemistry  and  Industry ,  Januaiy  1970. 

Continuation  of  Parker's  daacrlption  of  the  use  of  Delphi  at  the 
Hercules  Powder  Company.  Detailed  tables  and  charts  showing  study  re¬ 
sults  are  Included.  The  results  are  also  interpreted  In  the  text.  An 
appendix  reproduces  the  Third  Round  Questionnaire.  (B) 

PARKER,  E.  P. 

"Sosa  Experience  with  the  Application  of  the  Delphi  Method" 

Chemistry  and  Industry,  No,  38,  London,  September  1969,  pp.  1317-1319. 

This  is  a  general  review  of  Parker's  experience  with  Delphi  at  the 
Hercules  Powder  Company.  Some  basic  results  of  the  study  are  also  ref¬ 
erenced.  (B) 
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P ARSONS,  HENRY  MCXLVA1NZ 

Man-Machine  System  Experiments 

The  John  Hopkins  Press,  Baltimore,  Maryland ,  1972. 

Nan-machine  ays tea  experiaents  comprise  e  field  of  research  which 
has  developed  in  the  lest  two  decades  in  response  to  new  technologies 
such  as  radar  end  coaputers.  Despite  the  aegnitude  end  inportance  of 
this  field,  little  ebout  its  methodology  hea  appeared  in  print,  nor  have 
reports  of  the  experiments  themselves  been  generally  accessible. 

In  contrast  to  other  human  factors  studies,  a  large-scale  man- 
machine  system  experiment  typically  investigates  coordinated  team  per¬ 
formance  that  may  involve  a  dozen  or  more  system  operators.  Although 
the  elaborate  simulation  required  if  often  computer-based,  in  this 
reseat,  'i — unlike  all-computer  simulations — actual  people  operate  actual 
machines.  For  some  experiments  major  facilities  have  been  created;  for 
others,  the  system  lteslf  has  been  the  laboratoiy. 

This  book  describes  more  than  200  of  these  complex  experiments  and 
the  forty-odd  programs  in  which  they  occurred  et  universities,  non¬ 
profit  institutions,  and  government  research  agencies.  The  author  has 
drawn  on  information  from  some  600  references,  numerous  investigators 
and  consultants,  and  first-hand  experience  to  prone  what  he  once  called 
"fifty  million  dollars  of  burled  research." 

He  also  analyzes  systematically  Lhe  methodological  problems  that 
result  from  experlswntatlon  on  air  defense  systems,  air  traffic  control 
systems,  logistics  organizations,  space  flight,  battlefield  operations, 
police  dispatching,  and  even  communications  between  heads  of  state.  He 
includes  experimental  results  concerning  system  procedures,  training 
methods,  team  composition,  organizational  adaptation,  complex  decision¬ 
making,  man-machine  capabilities,  and  many  aspects  of  design  including 
degrees  of  computer  automation. 

This  comprehensive  survey  can  guide  those  who  plan  similar  exper¬ 
iments  in  the  future.  It  will  Interest  psychologists  and  engineers 
engaged  in  human  factors  research  and  application,  data  processing  and 
operations  research  specialists  in-  system  analysis  and  simulation,  tea¬ 
chers  of  the  experimental  method,  and  all  those  concerned  with  the  roles 
of  man  and  machine  in  today's  fast-developing  technologies. 

PARSONS  and  WILLIAMS 

Forecast  1968-2000  of  Computer  Developments  and  Applications 

Copenhagen,  1968. 

This  Delphi  study  was  prepared  by  Parsons  and  Williams  (Danish 
consultants)  as  an  input  to  panel  discussions  at  a  conference  sponsored 
by  the  International  Federation  of  Information  Processing  Societies. 

A  wide  vsrlety  of  computer  applications  and  technological  advances  are 
explored.  Study  respondents  were  individuals  planning  to  attend  the 
conference.  (B) 

PHILLIPS  PETROLEUM 

Participants  in  a  study  and  users  of  Delphi  output  in  a  multi- 
industry  study  described  by  GLAZIER  et  al . 
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PILL,  JURI 

"The  Delphi  Method:  Substance,  Contexts,  A  Critique  and  an  Annotated 
Bibliography" 

Socio-Economic  Planning  Sciences,  Vol.  5,  1971,  pp.  57-71. 

While  this  paper  only  makes  passing  reference  to  Industrial  Delphi 
work  (i.e.,  TRW),  It  does  contain  an  excellent  annotated  bibliography 
on  the  technique  itself.  (B) 

PWG  PUBLICATIONS 

"Japanese  Futures  Research  Trend" 

Technology  Forecaste  and  Technology  Surveys,  June  1973,  pp.  2-3. 

PYKE,  DONALD  L. 

"Mapping — A  System  Concept  for  Displaying  Alternatives" 

A  Guide  to  Practical  Technological  Forecasting ,  James  R.  Bright  and 
M.  E.  F.  Schoeman,  (eds.),  Prentice-Hall,  Inc.  Englewood  Cliffs,  New 
Jersey,  1973. 

Pyke  elaborates  further  on  the  TRW  experience  with  Delphi.  The 
emphasis  In  this  paper  Is  on  the  use  of  the  forecasts  from  the  PROBF.  II. 
Pyke  shows  how  the  various  forecasts  can  be  considered  in  relationship 
to  each  other  through  a  mapping  technique  (a  FERT  type  chart).  (B) 

PYKE,  DONALD  L. 

"Technological  Forecasting:  A  Framework  for  Consideration" 

Futures,  Vol.  2,  No.  4,  December  1970,  pp.  327-331. 

PYKE.  D.  L. 

"Practical  Approaches  to  Delphi" 

Futures,  Vol.  2,  No,  2,  June  1970,  pp.  143-152. 

QUADE,  E.  S. 

On  the  Limitations  of  Quantitative  Analysis 

The  Rand  Corporation,  P-4530,  December  1970. 

Today  we  often  hear  that  to  meet  the  many  challenges  to  our  society 
we  need  only  turn  to  systems  analysis,  operations  research,  "aerospace 
technology" — In  other  words,  to  supposedly  objective  quantitative  an¬ 
alysis.  Few  realize  that  quantitative  analysis  ultimately  reduces  to 
an  orderly  series  of  judgments  in  defining  the  problem,  selecting  hy¬ 
potheses  and  approaches,  making  assumptions,  determining  the  "facts," 
assigning  numerical  values  and  relationships.  Unfortunately,  the  cost/ 
benefit  criteria  usually  differ  greatly  from  those  used  by  political 
participants.  Operational  gaming  with  role-playing  allows  fer  the  in¬ 
terplay  of  different  viewpoints  and  for  making  decisions  in  context  as 
the  need  arises.  For  situations  still  harder  to  model,  expert  judgment 
is  commonly  sought.  Delphi  questioning,  featuring  anonymity,  iteration, 
controlled  feedback,  and  statistical  response,  is  increasingly  used, 
but  more  experimental  work  is  needed.  Viewed  as  a  method  for  investi¬ 
gating  problems  rather  than  solving  them,  analysis  is  nearly  always 
useful.  (R) 

QUADE,  E.  S. 

An  Extended  Concept  of  "Model" 

The  Rand  Corporation,  P-4427,  July  1970. 
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A  broad  vlaw  of  tha  traditional  oparatlons  nodal!  suggesting  that 
any  davlce  b«  accepted  aa  a  model  If  it  providaa  a  logical  means  of  pre¬ 
dicting  and  comparing  the  outcomaa  of  alternative  act  Iona,  ragardleee 
of  ita  repreaentatlve  featurea  or  ita  efficiency  In  optimisation.  Il¬ 
lustrative  of  this  extended  type  of  model  la  Delphi,  an  Iterative  pro¬ 
cedure  for  eliciting  and  refining  the  opinions  of  a  group  of  people 
by  maans  of  a  series  of  questionnaires.  While  much  remains  to  be  learned 
about  Delphi  and  the  use  of  expertise,  its  potential  is  considerably 
wider  than  published  applications  indicate.  Industrial  and  urban  plan¬ 
ners,  research  managers,  and  policymakers  have  shown  extensive  interest. 
Suggested  applications  range  from  the  drafting  of  diplomatic  notes  and 
long-range  political  forecasting  to  determining  what  products  to  market. 
Because  it  can  be  used  to  allocate  resources  rationally  and  to  force 
explicit  thinking  about  the  measurement  of  benefits,  Delphi  offers  a 
hope  of  introducing  cost-effectiveness  thinking  to  s  wide  range  of  prob¬ 
lems  where  conventional  models  are  difficult  to  formulate.  (R) 

QUADE,  E.  S. 

Cost-Effectiveness:  Some  Trends  in  Analysis 

The  Rand  Corporation,  P-3529-1,  March  1970. 

Four  trends  can  be  seen  in  the  development  of  cost-effectiveness 
analysis.  (1)  Close  man-computer  interaction — through  on-line,  time- 
shared  systems  with  individual  consoles,  natural  language,  graphic  in¬ 
put,  and  disc-stored  submodels — enables  the  user  to  modify  his  program 
instantly,  even  during  execution.  Hence,  it  facilitates  the  numerous 
parametric  investigations  and  sensitivity  analyses  necessary  in  ranking 
alternatives.  (2}  The  theory  of  n-pereoo  games  accommodates  the  models 
with  many  Insignificant  playara  so  common  In  cost-effsctlvenass  analyses. 
Judgment  and  intuition  guide  both  quantitative  and  nonquant it at ive  as¬ 
pects  of  an  analyaie,  especially  those  with  high  social  and  political 
content.  (3)  Expertise  can  be  systematically  exploited  by  Delphi,  an 
iterative  procedure  for  eliciting  end  refining  group  opinions  through 
a  series  of  questionnaires  with  anonymous  response.  (4)  Analysts  ere 
now  facilitating  Implementation  of  their  studies  by  Investigating  the 
potential  effectiveness,  political  feasibility,  the  Internal  organisa¬ 
tional  acceptability  of  thalr  recommendations,  end  the  relationships 
between  individual  and  organisational  needs  and  objectives.  (R) 

QUINN,  JAMES  BRIAN 
"Technological  Forecasting" 

Harvard  Business  Review  Foreoast  Series,  No.  21215  (1971),  pp.  51-53. 

REISMAN,  A.,  S.  J.  MANTEL,  JR.,  B.  V.  DEAN,  and  N.  EISENBERG 
Evaluation  and  Budgeting  Model  for  a  System  of  Social  Agencies 
Department  of  Operations  Research,  Cess  Western  Reserve  University, 
Technical  Memorandum  No.  167,  1969. 

This  report  describee  a  relatively  large-scale  experiment  on  scal¬ 
ing  group  opinion  using  a  procedure  based  essentially  on  Delphi,  in 
that  it  used  controlled  feedback  end  anonymity.  The  main  difference 
was  that  an  overt  attempt  was  mads  to  schlsve  consensus,  rather  then 
stopping  at  a  prescribed  number  of  rounds. 
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The  objective  of  tha  study  was  to  estimate  the  relative  values  of 
aervlcee  performed  by  the  agencies  of  the  Jewish  Community  Federation 
(JCF)  of  Cleveland,  so  perceived  by  the  members  of  that  community. 

There  was  a  test  run  using  members  of  the  research  staff  who  were  them¬ 
selves  members  of  the  community,  then  a  procedure  using  the  JCF  staff, 
and  finally  a  panel  of  community  lay  leaders.  There  was  a  ranking  of 
250  "client-service  packages."  Correlation  was  surprisingly  good  among 
the  three  groups.  The  experiment  demonstrated  the  feasibility  of  ob¬ 
taining  a  set  of  meaningful  community  standards ,  meaningful  In  the  sense 
that  the  representatives  of  the  community  accepted  them  ae  valid  and 
raallatlc.  Tha  process  also  proved  to  have  value  In  Itself  aa  a  com¬ 
munication  device,  and  to  make  It  easier  to  use  the  results,  since  the 
participants  had  helpad  generate  them.  (P) 

REIS  MAN,  A.,  and  M.  I.  TAFT 

Computer  Time-Saving  Applioatione  in  Economic  Analysis :  An  Integrated. 
Approach 

Department  of  Operations  Resaarch,  Case  Western  Reserve  University, 
Tschnical  Memorandum  No.  1S1,  1969. 

A  generalised  evaluation  program  known  as  EVAL,  and  a  general 
"present  worth"  program  known  as  CERBS  have  been  developed  In  the  FOR¬ 
TRAN  IV  Computer  language  and  are  currently  operational  on  a  time¬ 
sharing  or  batch  basis.  Tha  programs  lead  an  individual  or  a  group  of 
people  through  a  systematic  step-by-step  procedure  for  evaluating  the 
relative  value  (utility  Including  present  worth)  of  each  of  a  given  set 
of  alternatives  with  respect  to  explicitly  stated  goals  and  objectives. 
Applications  In  auch  erase  as  equipment  design,  selection,  optimization 
end  replacement;  personnel  administration;  end  allocation  of  resources 
ere  presented.  (P) 

RKISMAN,  A.,  end  M.  I.  TAFT 

On  a  Computer-Aided  System  Approach  to  Personnel  Adninie tration 

Department  of  Operations  Research,  Case  Western  Reserve  University, 
Technical  Memorandum  No.  147,  1968. 

A  systematic  evaluation  methodology  has  bean  developed  which  Inte¬ 
grates  eoma  major  concspts  from  value,  utility,  decision,  aubjective 
probability  theorise  and  the  Delphi  method  for  obtaining  e  consensus  of 
opinions.  Thssa  theorise  ere  applied  to  the  process  of  evaluation  of 
personnel  for  recruitment,  promotions,  merit  raises,  transfer,  salary 
administration,  training  and  dsvelopmsnt.  The  model  requires  end  util¬ 
ises  ee  Inputs  explicitly  stated  sets  of  long-range  goals,  short-range 
objectives,  rssource  needs,  evaluative  criteria,  weighting  and  utility 
functions,  as  well  aa  tha  aubjactlve  Judgaants  of  appropriate  evalua¬ 
tors.  Tha  procaselng  of  this  information  may  he  Implemented  by  manual 
calculations,  batch  processing  on  an  IBM  1620  computer,  or  by  direct 
simulation  on  a  large  tlme-aharlng  computer  system.  By  utilising  stand¬ 
ard  statistical  procedures  end  the  decision  rule  to  maximise  expected 
utility,  the  methodology  produces  the  type  of  output  information  re¬ 
quired  for  rational  decisionmaking.  (P) 

RESCUER,  N. 

Delphi  and  Values 

The  Rend  Corporation,  P-4182,  September  1969. 
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An  analysis  of  the  utility  of  the  Delphi  method  for  determining  group 
values.  Delphi,  a  process  for  eliciting  group  opinion  by  a  series  of 
questionnaires  with  selective  feedback  of  earlier  responses,  has  tra¬ 
ditionally  bean  used  for  assessing  expert  opinion  about  factual  ques¬ 
tions.  In  the  area  of  values,  Delphi  is  more  appropriate  for  exploring 
group  evaluation  (which  involves  rational  criteria)  than  group  value 
judgment  (which  depends  on  emotional  response).  Delphi  can  also  be  an 
instrument  for  (1)  determining  the  values  most  relevant  to  the  decision 
by  focusing  upon  the  reasons  (and  their  relative  weights)  for  making 
certain  choices;  (2)  discovering  areas  of  consensus  within  general  value 
conflicts;  (3)  finding  subgroups  of  variant  opinion;  (4)  resolving  con¬ 
flicts  of  interest  by  questioning  noninvolved  third  parties.  However, 
the  utility  of  Delphi  for  ascertaining  values  is  limited  by  the  impos¬ 
sibility  of  checking  the  group  value  judgments  against  "the  actual 
facts."  (R) 

RESCHER,  N. 

The  Future  as  an  Object  of  Research 

The  Rand  Corporation,  P-3S93,  1967 

Limits  "the  perspective  to  two  sectors  of  particular  interest  alike 
to  makers  of  public  policy  and  to  reflective  citizens  in  the  modern 
world:  science  and  technology  on  the  one  hand,  and  our  human  and  social 
environment  upon  the  other." 

It  is  now  in  fashion  to  speculate  about  the  future.  The  Futures 
Industry.  The  problem  of  predictive  methodology.  Some  major  difficul¬ 
ties  for  prediction:  feedback,  change  occurrence,  fashions,  values, 
the  problems  of  data. 

This  is  an  excellent  summary  of  the  future  as  an  object  of  research. 
(P) 

ROCHBERG,  R. 

Some  Convents  on  the  Problems  of  Self-Affecting  Prediction 

The  Rand  Corporation,  P-3735,  1967. 

This  paper  attempts  to  analyze  and  clarify  the  possible  Interaction 
of  prediction  and  event.  (1)  Is  the  Interaction  of  prediction  and  event 
predictable?  (2)  Can  the  interaction  of  prediction  and  event  be  con¬ 
trolled?  (3)  How  and  to  what  extent  should  people  making  predictions 
try  to  take  into  account  the  possible  effects  of  their  predictions? 

(4)  Is  the  entire  phenomenon  of  self-affecting  predictions  of  any  prac¬ 
tical  use? 

He  uses  flow  charts,  a  probabilistic  model,  a  cybernetic  model  and 
discusses  various  types  of  prediction.  In  general,  this  is  a  good  over¬ 
view  of  the  problem,  without  any  real  theory  of  answers.  (P) 

ROGERS,  C. 

Encounter  Groups 

First  Edition,  Harper  and  Row,  New  York,  1970. 

ROGERS,  C, 

On  Becoming  a  Person 

Houghton  Mifflin,  Boston,  1961. 
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ROSOVE,  PERRY  E. 

The  Use  of  Contextual  Mapping  to  Support  Tong-Range  Educational  Policy 

Making 

SP-3026,  Syiteo  Development  Corporation,  Santa  Monica,  California,  De¬ 
cember  1967. 

SACKMAN,  H.  and  R.  CITRRNBAUM  (Eda) 

Online  Planning 

Prentice-Hall,  Englewood  Cliffs,  Nes  Jersey,  1972. 

SACKMAN,  H. 

An  Investigation  of  Certain  Aspects  of  the  Validity  of  the  Formal  Ror¬ 
schach  Scoring  System  in  Relation  to  Age,  Education,  and  Vocalndary 
Score 

Unpublished  doctoral  dissertation,  Fordham  University,  1952. 

SAHR,  ROBERT  C. 

A  Collation  of  Similar  Delphi  Forecasts 

Institute  for  the  Future,  WP-5,  April  1970. 

A  comparison  of  judgments  made  by  different  Delphi  panels  can  have 
two  main  purposes.  Firat,  it  can  seek  to  illuminate  differences  of  per¬ 
spective  among  disciplines.  Second,  it  can  help  identify  differences 
that  arise,  not  from  panel  assessments  and  Judgments,  but  from  extra¬ 
neous  factors.  Judicious  use  of  such  comparisons  can  help  to  refine 
Delphi  methodology  ao  that  its  results  are  more  reliable  reflections 
of  its  goals — eliciting  expert  Judgment. 

Because  the  Delphi  studies  discussed  here  were  not  designed  for 
comparison,  this  work,  as  the  title  statea,  ie  a  very  preliminary  one. 
The  author  hopea  that  che  approaches  used  here  may  suggest  ways  in 
which  Delphi  studies  may  be  designed  and  conducted  in  the  future  so  as 
to  facilitate  comparison. 

The  research  presented  in  this  paper  was  sponsored  by  the  Educa¬ 
tional  Policy  Research  Center  of  the  Syracuse  University  Research  Cor¬ 
poration. 

SALANCIK,  G,  R. ,  T.  J.  GORDON,  and  N.  ADAMS 

On  the  Nature  of  Economic  Losses  Arising  from  Computer-Based  Systems 
in  the  Next  Fifteen  Years 

Institute  for  the  Future,  R-25,  March  1972. 

This  study,  sponsored  by  the  Skandia  Insce.  Co.  explores  potential 
new  uses  of  computers  In  the  next  fifteen  years  and  the  potential  losses 
they  could  present  in  the  event  of  misuse  and  malfunction.  The  report 
examines  future  growth  of  computer  usage,  risks  attends  it  upon  likely 
future  computer  applications  and  possible  remedies  for  these  risks. 

(B) 

SALANCIK,  G,  R. ,  W,  WENGER,  and  E.  HELFER 
"The  Construction  of  Delphi  Statements" 

Technological  Forecasting  and  Social  Change ,  Vol.  3,  No.  1,  1971. 
SCHMIDT,  D.  L. 

Creativity  in  Industrial  Engineering 

The  Rand  Corporation,  P-4601,  M*.rch  1971. 
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A  dlscuBBion  of  creativity  and  Delphi  techniques  to  Improve  company 
research.  The  creative  process  synthesises  knowledge,  logical  reason¬ 
ing,  and  originality — the  mix  depending  on  the  field.  In  addition,  the 
creative  person  uses  multiple  approaches,  self-questioning,  scepticism, 
nonconformity,  and  self-confidence  and  courage  in  presenting  his  ideas. 
Management  techniques  to  motivate  creativity  are  many,  such  as  gearing 
tht  pressure  to  the  goals,  showing  tolerance  for  failure,  providing 
means  of  comuivicatlon  with  colltcguae,  csttlng  a  creative  atmosphere, 
welcoming  disagreement,  end  accepting  unconventionality.  The  Delphi 
procedure,  by  encouraging  anonymity,  controlled  feedback,  and  statisti¬ 
cal  "group  response"  reduces  the  undesirable  socially  dominant  and  group 
pressure  toward  conformity.  Delphi  also  includes  some  form  of  self- 
rating.  In  conjunction  with  the  uetaMlshnwnt  of  an  environment  and  a 
management  team,  the  Delphi  technique  could  considerably  improve  the 
creative  output  of  a  company's  research  staff.  (R) 

SET  INC. 

See  DELPHI  PANE  I  ON  THE  FUTURE  OF  LEISURE  AND  RECREATION,  SE1  INC. 
SHAM,  M. 

Group  Dynamics:  The  Psychology  of  Smalt  Croup  Behavior 

McCraiw  Hill,  New  York,  1971. 

SHERI F,  M. 

The  Psychology  of  Social  Nome 

Harper,  New  York,  1936. 

SILLS,  DAVID  L.  (Ed.) 

International  Encyclopedia  of  the  Social  Sciences 

Macmillan,  New  York,  1968. 

SKANDIA  INSCE  CO. 

Sponsor  of  the  Institute  for  the  Future  study  R-25:  On  the  Nature 
of  Eaonomia  Losses  Arising  from  Computer-Based  Services  in  the  Next 
Fifteen  Years,  (SALANCIK,  GORDON,  and  ADAMS). 

SMIL,  V. 

"Energy  and  the  Environment — A  Delphi  Forecast" 

Long-Bange  Planning ,  Vol.  5,  No.  4,  December  1972,  pp.  27-32. 

SMITH,  KLINE,  AND  FRENCH  LABORATORIES 

See  papers  by  BENDER  and  TEELING-SMITH.  The  Bender  article  gives 
detailed  results  from  the  study  in  graphical  form.  Also  refsrsncsd  in 
Business  Week  article,  MARTINO,  Technological  Forecasting ,  and  CETRON'S 
book.  (B) 

STEINER,  GEORGE  A. 

Top  Management  Planning 

The  Macmillan  Company,  London,  1969. 

STOGDILL,  R.  M. 

Individual  Behavior  and  Group  Achievement 

Oxford  University  Press,  New  York,  1959. 
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SULC,  0. 

Forecasting  the  Internationa  Between  Technological  and  Si>cial  Changes 

Manchester  Business  School,  Univereity  of  Manchester,  06I.-237-4571, 

July  1968, 

TEELING-SM1TH ,  GEORGE 

"Medicines  In  the  1990' s:  Experience  with  a  Delphi  Forecast" 

Long-Range  Planning ,  June  1971. 

Describes  a  Delphi  In  the  Office  of  Health  Economics,  London. 
References  U.S.  Delphi  studies  by  Smith,  Kline  and  French  plus  three 
unnamed  other  U.S.  companies.  (B) 

"The  Futures  Business" 

Chemical  4  Engineering  News,  Vol.  47,  August  1969,  pp.  62-75. 

This  article  reviews  the  rapid  growth  of  long-term  forecasting 
activities  In  business,  government,  and  policy  research  organizations. 
Recent  (to  1969)  books,  journal  articles,  and  studies  are  referenced. 
The  examination  of  Delphi  reviews  the  Rand  and  TRW  e::perlences .  One 
chart  of  TRW  pi  edict  Ions  Is  shown.  (B) 

THIESMEYBR,  LINCOLN  R. 

"How  to  Avoid  Bandwagon  Effect  in  Forecasting:  The  Delphi  Conference 
Keeps  Crystal  Ball  Clear" 

Financial  Post ,  October  1971. 

A  general  review  of  the  Delphi  technique.  The  Institute  for  the 
Future  Delphi  studies  are  referenced.  (B) 

THIESMEYBR,  LINCOLN  R. 

"The  Art  of  Forecasting  the  Future  is  Losing  Its  Amateur  Status" 
Financial  Post,  June  1971,  p.  7. 

Describes  the  growth  of  "futurology"  and  technological  forecasting 
in  business  and  the  creation  of  the  Institute  for  the  Future  with  cor¬ 
porate  help.  References  IFF  Delphi  on  residential  housing  sponsored  by 
Owens-Coming  Flberglas  Corporation.  Also  mentions  activities  at  Gen¬ 
eral  Electric,  Goodyear  Tire  and  Rubber,  and  Delphi  by  Smith,  Kline  and 
French:  "A  Delphi  Study  of  the  Future  of  Medicine."  (B) 

THOMPSON,  L.  T. 

A  Pilot  Application  of  Delphi  Techniques  to  the  Drug  Field:  Some  Ex¬ 
perimental  Findings 

The  Rand  Corporation,  R-1124,  June  1973. 

Report  on  an  experiment  designed  to  estimate  the  feasibility  and 
usefulness  of  employing  Delphi  techniques  to  examine  drug-related  Is¬ 
sues.  The  experiment  consisted  of  two  Delphi  rounds  and  used  a  ques¬ 
tionnaire  which  Included  both  descriptive  and  evaluative  items  pertin¬ 
ent  to  drug  use  and  drug  programs.  Drug-related  questions  are  partic¬ 
ularly  prone  to  ambiguity  end  interpretive  difficulties;  a  useful  side 
benefit  of  Delphi  In  this  context  is  the  capacity  to  modify  or  augment 
questions  on  the  basis  of  first-round  responses.  Although  some  conver¬ 
gence  behavior  was  observed  on  almost  all  questions,  the  degree  of  con¬ 
vergence  differed  significantly  among  items. 


THORNDIKE,  ROBERT  L. 

Personnel  Selection 

Wiley,  New  York,  1949. 
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TRANS  CANADA  TELEPHONE  SYSTEM 

Communications ,  Computers ,  and  Canada 

December  1971. 

This  report  by  the  TCTS  (8  major  Canadian  Telephone  Companies)  out¬ 
lines  their  commitment  to  long-range  planning  for  computer  and  visual 
communications  services.  Three  Bell  Canada  Business  Planning  Delphi 
studies  are  abstracted  in  Appendix  B.  Sample  results  are  shown  from 
the  Education,  Medicine,  and  Business  studies  (see  DOYLE  and  GOODWILL, 
and  GOODWILL  references).  (B) 

TRW  DELPHI  STUDIES 

See  works  by  NORTH  and/or  PYKE.  Results  also  shown  in  WILLS: 
Technological  forecasting  and  CETRON’s  book.  Many  of  the  other  arti¬ 
cles  in  this  bibliography  have  brief  references  to  the  TRW  r _iphi  work. 
Footnote  No.  16  lists  several  other  references  not  outlined  in  the  bib¬ 
liography.  (B) 

TUROFF,  M. 

"An  Alternative  Approach  to  Cross-Impact  Analysis" 

Technological  Forecasting  and  Social  Change,  Vol.  3,  No.  3,  1972,  pp. 
309-339.' 

Reviews  the  literature  on  the  subject  and  the  use  of  cros9-impact 
in  an  information  system  context. 

TUROFF,  M. 

"Delphi  Conferencing  (i.e.,  Computer  Based  Conferencing  with  Anonymity)" 

'Technological  Forecasting  and  Social  Change,  Vol.  3,  No.  2,  1972,  pp. 
159-204. 

TUROFF,  M. 

'Jhe  Design  of  a  Policy  Delphi 

National  Resource  Analysis  Center,  Systems  Evaluation  Division,  Execu¬ 
tive  Office  of  the  President,  Office  of  Emergency  Preparedness,  T.K. 

123,  1970.  (See  also  Technological  Forecasting  and  Social  Change,  Vol. 
2,  No.  2,  1970,  pp.  149-171.) 

This  paper  is  a  comprehensive  and  definitive  examination  of  the 
use  of  the  Delphi  method  in  policy  issues.  It  develops  the  thesis  that 
the  Delphi  could  be  useful  as  an  adjunct  to  the  committee  approach  in 
policy  formulation;  it  is  argued  that  the  two  are  complimentary,  rather 
than  one  being  superio 

The  paper  opens  wich  a  review  of  Delphi,  argues  for  its  use  in 
policy  decisions  while  pointing  out  possible  dangers,  and  recommends 
some  procedural  guidelines. 

There  is  a  highly  complete  and  up-to-date  bibliography  included. 

(PI 

TUROFF,  M. 

"Delphi  and  its  Potential  Impact  on  Information  Systems" 
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AFIP3  Conference  Proceedings ,  Vol.  39,  pp.  317-326,  AFIPS  Press,  Mont- 
vsle,  New  Jersey,  1971. 

VOYER,  R.  D. 

"Inventing  Future  with  Custommadc  Technology" 

Science  Forum ,  Vol.  3,  No.  5,  1970,  p.  8. 

VOYER,  R.  D. 

"Delphi  Technique— A  Valuable  Tool  for  Technological  Forecasting" 

Science  Forum,  Vol.  2,  No.  5,  1969,  p.  6. 

WEAVER,  W.  T. 

Delphi,  A  Critical  Review 

Syracuse  University  Research  Corporation,  RR-7,  February  1972. 

WEAVER,  W.  T. 

Delphi  as  a  Method  for  Studying  the  Future:  Testing  Some  Underlying 
Assumptions 

Educational  Policy  Research  Center,  Syracuse,  New  York,  1970. 

WEAVER  W.  T. 

An  Exploration  Into  the  Relationship  Between  Conceptual  Level  and  Fore¬ 
casting  Future  Events 

Doctoral  Dissertation,  Syracuse  University,  New  York,  1969. 

WEYERHAEUSER  COMPANY 

This  study  is  mentioned  in  the  Business  Week  article.  The  Delphi 
reseerch  is  into  the  future  of  the  construction  business.  (B) 

WHIRLPOOL 

See  the  paper  by  DAVIS  for  Whirlpool's  use  of  Delphi  as  a  part  of 
their  information  gathering  system.  (B) 

wilcox,  w. 

Forecasting  Asian  Strategic  Environments  for  National  Security  Decision¬ 
making:  A  Report  and  a  Method 

The  Rand  Corporation,  RM-6154,  June  1970. 

WILLS,  GORDON 

"The  Preparation  and  Development  of  Technological  Forecasts" 

Long-Range  Forecasting ,  March  1970. 

A  review  of  technological  forecasting  and  planning  in  business. 
Several  techniques  are  reviewed.  Delphi  results  from  a  computer  study 
by  ICL  are  shown.  The  Hercules  Powder  Company  study  is  referenced  as 
well  and  results  shown.  The  first  round  Hercules  questionnaire  is 
shown  lx.  an  appendix.  (B) 

WILLS,  GORDON,  RICHARD  WILSON,  NEIL  MANNING,  and  ROGER  HILDEBRANDT 

Technological  Forecasting 

Penguin  Books,  Middlesex,  England,  1972. 

This  book  subtitled  "The  Art  and  its  Managerial  Implications" 
examines  technological  forecaating  from  an  essentially  marketing  view¬ 
point.  The  chapter  on  Delphi  uaes  business  examples.  The  Hercules 


-118- 


study  Is  cited,  showing  the  first  round  questionnaire  (pp.  194-203) 
study  design,  and  sample  results  (pp.  209-210).  A  sample  feedback  from 
a  Llng-Tomco-Vaught,  Inc.  (LTV)  Study  la  also  shown  (p.  211).  Three 
charts  illustrate  the  ICL  study  (pp.  212-215).  The  chapter  concludes 
with  a  discussion  of  the  TRW  experience,  including  charts  (pp.  216-223). 
(B) 

ZARNOWITZ,  VICTOR 

"On  the  Accuracy  and  Properties  of  Short-Term  Economic  Forecasts" 

The  Task  of  Eoonotnies ,  Forth-Fifth  Annual  Report  of  the  National  Bureau 
of  Economic  Research,  1965. 


